Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250625 20:00:49.102861 816 test_util.cc:276] Using random seed: -423817220
2025-06-25T20:00:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T20:00:49Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasknmPaw0/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasknmPaw0/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/security.properties
2025-06-25 20:00:55: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasknmPaw0/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasknmPaw0/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250625 20:01:04.514101 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.62:36889
--webserver_interface=127.0.204.62
--webserver_port=0
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 20:01:04.618131 1060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:01:04.618376 1060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:04.618417 1060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:01:04.621632 1060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:01:04.621698 1060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:04.621719 1060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:01:04.621737 1060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:01:04.625475 1060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.62:36889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.204.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:04.626554 1060 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:04.627681 1060 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:01:04.634907 1068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:04.634907 1066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:04.635015 1060 server_base.cc:1048] running on GCE node
W20250625 20:01:04.634940 1065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:04.635640 1060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:04.636584 1060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:04.637724 1060 hybrid_clock.cc:648] HybridClock initialized: now 1750881664637689 us; error 50 us; skew 500 ppm
I20250625 20:01:04.648847 1060 webserver.cc:469] Webserver started at http://127.0.204.62:37479/ using document root <none> and password file <none>
I20250625 20:01:04.649632 1060 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:04.649703 1060 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:04.649914 1060 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:01:04.651592 1060 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/instance:
uuid: "5852fb53503b438c92f294e0c44c916d"
format_stamp: "Formatted at 2025-06-25 20:01:04 on dist-test-slave-lc57"
I20250625 20:01:04.652117 1060 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal/instance:
uuid: "5852fb53503b438c92f294e0c44c916d"
format_stamp: "Formatted at 2025-06-25 20:01:04 on dist-test-slave-lc57"
I20250625 20:01:04.655444 1060 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250625 20:01:04.657584 1074 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:04.658591 1060 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250625 20:01:04.658687 1060 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
uuid: "5852fb53503b438c92f294e0c44c916d"
format_stamp: "Formatted at 2025-06-25 20:01:04 on dist-test-slave-lc57"
I20250625 20:01:04.658776 1060 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:04.682883 1060 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:04.683543 1060 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:04.683744 1060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:04.691121 1060 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.62:36889
I20250625 20:01:04.691129 1126 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.62:36889 every 8 connection(s)
I20250625 20:01:04.692199 1060 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/info.pb
I20250625 20:01:04.694751 1127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:04.698144 1127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } has no permanent_uuid. Determining permanent_uuid...
I20250625 20:01:04.699783 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 1060
I20250625 20:01:04.699911 816 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal/instance
I20250625 20:01:04.701228 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.61:38005
--webserver_interface=127.0.204.61
--webserver_port=0
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 20:01:04.708894 1127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } has no permanent_uuid. Determining permanent_uuid...
W20250625 20:01:04.709676 1075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
W20250625 20:01:04.710482 1127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.204.61:38005: Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:04.759743 1127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } attempt: 1
W20250625 20:01:04.761196 1127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.204.61:38005: Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
W20250625 20:01:04.806213 1130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:01:04.806445 1130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:04.806500 1130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:01:04.810145 1130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:01:04.810227 1130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:04.810247 1130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:01:04.810264 1130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:01:04.814028 1130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.61:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.204.61
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:04.815092 1130 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:04.816254 1130 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:01:04.822721 1139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:04.822721 1136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:04.822731 1137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:04.822844 1130 server_base.cc:1048] running on GCE node
I20250625 20:01:04.823386 1130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:04.824324 1130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:04.825515 1130 hybrid_clock.cc:648] HybridClock initialized: now 1750881664825494 us; error 33 us; skew 500 ppm
I20250625 20:01:04.827337 1130 webserver.cc:469] Webserver started at http://127.0.204.61:44189/ using document root <none> and password file <none>
I20250625 20:01:04.827940 1130 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:04.827997 1130 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:04.828168 1130 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:01:04.829370 1127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } attempt: 2
I20250625 20:01:04.829764 1130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/instance:
uuid: "f87f9e18e5b948a4a3fb3666f106abd0"
format_stamp: "Formatted at 2025-06-25 20:01:04 on dist-test-slave-lc57"
I20250625 20:01:04.830212 1130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal/instance:
uuid: "f87f9e18e5b948a4a3fb3666f106abd0"
format_stamp: "Formatted at 2025-06-25 20:01:04 on dist-test-slave-lc57"
W20250625 20:01:04.830574 1127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.204.61:38005: Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:04.833747 1130 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250625 20:01:04.836062 1145 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:04.837194 1130 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250625 20:01:04.837308 1130 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
uuid: "f87f9e18e5b948a4a3fb3666f106abd0"
format_stamp: "Formatted at 2025-06-25 20:01:04 on dist-test-slave-lc57"
I20250625 20:01:04.837410 1130 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:04.858057 1130 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:04.858749 1130 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:04.858923 1130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:04.867122 1130 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.61:38005
I20250625 20:01:04.867164 1197 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.61:38005 every 8 connection(s)
I20250625 20:01:04.868211 1130 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/info.pb
I20250625 20:01:04.870939 1198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:04.874552 1198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } has no permanent_uuid. Determining permanent_uuid...
I20250625 20:01:04.876892 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 1130
I20250625 20:01:04.876997 816 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal/instance
I20250625 20:01:04.877764 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.60:46191
--webserver_interface=127.0.204.60
--webserver_port=0
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 20:01:04.883983 1198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } has no permanent_uuid. Determining permanent_uuid...
I20250625 20:01:04.890136 1198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } has no permanent_uuid. Determining permanent_uuid...
W20250625 20:01:04.890892 1149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:01:04.891841 1198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.204.60:46191: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:04.921902 1127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } attempt: 3
I20250625 20:01:04.926415 1127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } has no permanent_uuid. Determining permanent_uuid...
W20250625 20:01:04.927788 1127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.204.60:46191: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:04.941084 1198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } attempt: 1
W20250625 20:01:04.942575 1198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.204.60:46191: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:04.959017 1127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } attempt: 1
W20250625 20:01:04.960458 1127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.204.60:46191: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:01:04.987059 1200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:01:04.987308 1200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:04.987347 1200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:01:04.990594 1200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:01:04.990669 1200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:04.990689 1200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:01:04.990705 1200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:01:04.994547 1200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.60:46191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.204.60
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:04.995625 1200 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:04.996804 1200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:01:05.003412 1210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:05.003417 1207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:05.003418 1208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:05.003485 1200 server_base.cc:1048] running on GCE node
I20250625 20:01:05.004145 1200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:05.005110 1200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:05.006312 1200 hybrid_clock.cc:648] HybridClock initialized: now 1750881665006287 us; error 33 us; skew 500 ppm
I20250625 20:01:05.008329 1200 webserver.cc:469] Webserver started at http://127.0.204.60:43997/ using document root <none> and password file <none>
I20250625 20:01:05.008912 1200 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:05.008970 1200 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:05.009145 1200 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:01:05.010761 1198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } attempt: 2
I20250625 20:01:05.010795 1200 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/instance:
uuid: "1fcdee0c0aac4013a5e3c77304b474d9"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
I20250625 20:01:05.011343 1200 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal/instance:
uuid: "1fcdee0c0aac4013a5e3c77304b474d9"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
W20250625 20:01:05.012032 1198 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.204.60:46191: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:05.014870 1200 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250625 20:01:05.017204 1216 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:05.018222 1200 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:05.018352 1200 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
uuid: "1fcdee0c0aac4013a5e3c77304b474d9"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
I20250625 20:01:05.018441 1200 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:05.035676 1127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } attempt: 2
W20250625 20:01:05.037029 1127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.204.60:46191: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:05.044574 1200 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:05.045249 1200 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:05.045421 1200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:05.053143 1200 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.60:46191
I20250625 20:01:05.053166 1268 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.60:46191 every 8 connection(s)
I20250625 20:01:05.054172 1200 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/info.pb
I20250625 20:01:05.056875 1269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:05.060266 1269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } has no permanent_uuid. Determining permanent_uuid...
I20250625 20:01:05.063421 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 1200
I20250625 20:01:05.063536 816 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal/instance
I20250625 20:01:05.069162 1269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } has no permanent_uuid. Determining permanent_uuid...
I20250625 20:01:05.073313 1269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } has no permanent_uuid. Determining permanent_uuid...
I20250625 20:01:05.081830 1269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Bootstrap starting.
I20250625 20:01:05.084075 1269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:05.084770 1269 log.cc:826] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:05.086632 1269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: No bootstrap required, opened a new log
I20250625 20:01:05.089429 1269 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:05.089608 1269 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:05.089650 1269 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcdee0c0aac4013a5e3c77304b474d9, State: Initialized, Role: FOLLOWER
I20250625 20:01:05.090018 1269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:05.090423 1277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:05.090544 1277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:05.091077 1269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:01:05.095208 1200 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: OK, instance_id { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" instance_seqno: 1750881664688748 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250625 20:01:05.097358 1200 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 20:01:05.099100 1291 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250625 20:01:05.099174 1291 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250625 20:01:05.103397 1198 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } attempt: 3
I20250625 20:01:05.110746 1198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Bootstrap starting.
I20250625 20:01:05.113741 1198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:05.114938 1198 log.cc:826] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:05.116781 1198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: No bootstrap required, opened a new log
I20250625 20:01:05.119192 1198 raft_consensus.cc:357] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:05.119403 1198 raft_consensus.cc:383] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:05.119480 1198 raft_consensus.cc:738] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87f9e18e5b948a4a3fb3666f106abd0, State: Initialized, Role: FOLLOWER
I20250625 20:01:05.119930 1198 consensus_queue.cc:260] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:05.120369 1295 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:05.120537 1295 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:05.121079 1198 sys_catalog.cc:564] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:01:05.125730 1130 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: OK, instance_id { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" instance_seqno: 1750881664688748 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250625 20:01:05.127911 1130 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 20:01:05.130658 1309 catalog_manager.cc:1560] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250625 20:01:05.130735 1309 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250625 20:01:05.136179 1127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } attempt: 3
I20250625 20:01:05.143615 1127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Bootstrap starting.
I20250625 20:01:05.146425 1127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:05.147614 1127 log.cc:826] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:05.165488 1127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: No bootstrap required, opened a new log
I20250625 20:01:05.168960 1127 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:05.169147 1127 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:05.169193 1127 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5852fb53503b438c92f294e0c44c916d, State: Initialized, Role: FOLLOWER
I20250625 20:01:05.169869 1127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:05.171252 1127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:01:05.181094 1060 master_runner.cc:263] No leader master found from master 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:05.181167 1060 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 20:01:05.185127 1080 tablet.cc:2378] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 20:01:05.200080 1327 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250625 20:01:05.200219 1327 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250625 20:01:05.200469 1313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:05.200591 1313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:05.281751 1277 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:05.281976 1277 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:05.283082 1277 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
I20250625 20:01:05.284286 1172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" is_pre_election: true
I20250625 20:01:05.284617 1172 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fcdee0c0aac4013a5e3c77304b474d9 in term 0.
I20250625 20:01:05.284756 1101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5852fb53503b438c92f294e0c44c916d" is_pre_election: true
I20250625 20:01:05.285034 1101 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fcdee0c0aac4013a5e3c77304b474d9 in term 0.
I20250625 20:01:05.285575 1217 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0; no voters:
I20250625 20:01:05.286314 1277 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:05.286391 1277 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:05.286427 1277 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:05.287549 1277 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:05.287968 1277 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 1 election: Requested vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
I20250625 20:01:05.288949 1172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0"
I20250625 20:01:05.289079 1172 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:05.290279 1172 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcdee0c0aac4013a5e3c77304b474d9 in term 1.
I20250625 20:01:05.291626 1217 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0; no voters:
I20250625 20:01:05.292201 1101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5852fb53503b438c92f294e0c44c916d"
I20250625 20:01:05.292346 1101 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:05.293519 1101 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcdee0c0aac4013a5e3c77304b474d9 in term 1.
I20250625 20:01:05.294088 1277 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:05.294379 1277 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 1 LEADER]: Becoming Leader. State: Replica: 1fcdee0c0aac4013a5e3c77304b474d9, State: Running, Role: LEADER
I20250625 20:01:05.294802 1277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
W20250625 20:01:05.296599 1222 tablet.cc:2378] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:01:05.296883 1277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fcdee0c0aac4013a5e3c77304b474d9. Latest consensus state: current_term: 1 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:05.296983 1277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:05.306007 1332 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:01:05.316731 1332 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:01:05.322259 1101 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 1 FOLLOWER]: Refusing update from remote peer 1fcdee0c0aac4013a5e3c77304b474d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 20:01:05.323035 1172 raft_consensus.cc:1273] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 1 FOLLOWER]: Refusing update from remote peer 1fcdee0c0aac4013a5e3c77304b474d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 20:01:05.322986 1330 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:05.323673 1330 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:05.326946 1295 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fcdee0c0aac4013a5e3c77304b474d9. Latest consensus state: current_term: 1 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:05.327068 1295 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:05.327234 1313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fcdee0c0aac4013a5e3c77304b474d9. Latest consensus state: current_term: 1 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:05.327356 1313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:05.331607 1277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:05.331759 1277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:05.331851 1333 mvcc.cc:204] Tried to move back new op lower bound from 7171611301152903168 to 7171611301051711488. Current Snapshot: MvccSnapshot[applied={T|T < 7171611301152903168}]
I20250625 20:01:05.332216 1313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:05.332334 1313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:05.332564 1334 mvcc.cc:204] Tried to move back new op lower bound from 7171611301152903168 to 7171611301051711488. Current Snapshot: MvccSnapshot[applied={T|T < 7171611301152903168}]
I20250625 20:01:05.333716 1332 catalog_manager.cc:1349] Generated new cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:05.333778 1332 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:01:05.336402 1330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:05.336521 1330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:05.337633 1295 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:05.337744 1295 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:05.339357 1335 mvcc.cc:204] Tried to move back new op lower bound from 7171611301152903168 to 7171611301051711488. Current Snapshot: MvccSnapshot[applied={T|T < 7171611301152903168}]
I20250625 20:01:05.357919 1332 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 20:01:05.359133 1332 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:01:05.369304 1332 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Generated new TSK 0
I20250625 20:01:05.370231 1332 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:01:05.370896 1332 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 20:01:05.371337 1332 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250625 20:01:05.393743 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.204.1:0
--local_ip_for_outbound_sockets=127.0.204.1
--webserver_interface=127.0.204.1
--webserver_port=0
--tserver_master_addrs=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250625 20:01:05.511085 1342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:01:05.511307 1342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:05.511358 1342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:01:05.514510 1342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:05.514633 1342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.204.1
I20250625 20:01:05.518357 1342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.204.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.204.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--never_fsync=true
--heap_profile_path=/tmp/kudu.1342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.204.1
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:05.519397 1342 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:05.520578 1342 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:01:05.528189 1347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:05.528236 1350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:05.528189 1348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:05.528525 1342 server_base.cc:1048] running on GCE node
I20250625 20:01:05.528918 1342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:05.529482 1342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:05.530682 1342 hybrid_clock.cc:648] HybridClock initialized: now 1750881665530653 us; error 50 us; skew 500 ppm
I20250625 20:01:05.532728 1342 webserver.cc:469] Webserver started at http://127.0.204.1:35475/ using document root <none> and password file <none>
I20250625 20:01:05.533270 1342 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:05.533337 1342 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:05.533524 1342 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:01:05.535131 1342 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/data/instance:
uuid: "27aa5b53b16747f9a72485aaf2417b08"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
I20250625 20:01:05.535602 1342 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/instance:
uuid: "27aa5b53b16747f9a72485aaf2417b08"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
I20250625 20:01:05.538993 1342 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20250625 20:01:05.541388 1356 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:05.542469 1342 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:05.542582 1342 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal
uuid: "27aa5b53b16747f9a72485aaf2417b08"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
I20250625 20:01:05.542668 1342 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:05.565392 1342 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:05.566066 1342 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:05.566233 1342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:05.566781 1342 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:01:05.567842 1342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:01:05.567891 1342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:05.567942 1342 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:01:05.567975 1342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:05.577294 1342 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.1:35759
I20250625 20:01:05.577308 1469 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.1:35759 every 8 connection(s)
I20250625 20:01:05.578270 1342 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/data/info.pb
I20250625 20:01:05.580621 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 1342
I20250625 20:01:05.580724 816 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/instance
I20250625 20:01:05.584070 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.204.2:0
--local_ip_for_outbound_sockets=127.0.204.2
--webserver_interface=127.0.204.2
--webserver_port=0
--tserver_master_addrs=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250625 20:01:05.589681 1470 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:05.589948 1470 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:05.590490 1471 heartbeater.cc:344] Connected to a master server at 127.0.204.62:36889
I20250625 20:01:05.590584 1471 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:05.590617 1470 heartbeater.cc:507] Master 127.0.204.60:46191 requested a full tablet report, sending...
I20250625 20:01:05.590785 1471 heartbeater.cc:507] Master 127.0.204.62:36889 requested a full tablet report, sending...
I20250625 20:01:05.591193 1472 heartbeater.cc:344] Connected to a master server at 127.0.204.61:38005
I20250625 20:01:05.591320 1472 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:05.591519 1472 heartbeater.cc:507] Master 127.0.204.61:38005 requested a full tablet report, sending...
I20250625 20:01:05.592026 1233 ts_manager.cc:194] Registered new tserver with Master: 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:05.592043 1091 ts_manager.cc:194] Registered new tserver with Master: 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:05.592527 1162 ts_manager.cc:194] Registered new tserver with Master: 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:05.593465 1233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.204.1:42881
W20250625 20:01:05.702689 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:01:05.702904 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:05.702956 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:01:05.707442 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:05.707602 1479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.204.2
I20250625 20:01:05.713411 1479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.204.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.204.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--never_fsync=true
--heap_profile_path=/tmp/kudu.1479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.204.2
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:05.714569 1479 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:05.715775 1479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:01:05.723915 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:05.724073 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:05.723927 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:05.724187 1479 server_base.cc:1048] running on GCE node
I20250625 20:01:05.724560 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:05.725145 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:05.726330 1479 hybrid_clock.cc:648] HybridClock initialized: now 1750881665726304 us; error 39 us; skew 500 ppm
I20250625 20:01:05.728336 1479 webserver.cc:469] Webserver started at http://127.0.204.2:41189/ using document root <none> and password file <none>
I20250625 20:01:05.728888 1479 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:05.728937 1479 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:05.729099 1479 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:01:05.730702 1479 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/data/instance:
uuid: "c73846c0665d4d53a274d053946107df"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
I20250625 20:01:05.731215 1479 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/instance:
uuid: "c73846c0665d4d53a274d053946107df"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
I20250625 20:01:05.734720 1479 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250625 20:01:05.737123 1493 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:05.738286 1479 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:05.738456 1479 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal
uuid: "c73846c0665d4d53a274d053946107df"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
I20250625 20:01:05.738582 1479 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:05.781883 1479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:05.782595 1479 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:05.782771 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:05.783334 1479 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:01:05.784400 1479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:01:05.784447 1479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:05.784490 1479 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:01:05.784523 1479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:05.794189 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.2:43561
I20250625 20:01:05.794206 1606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.2:43561 every 8 connection(s)
I20250625 20:01:05.795197 1479 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/data/info.pb
I20250625 20:01:05.801303 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 1479
I20250625 20:01:05.801434 816 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/instance
I20250625 20:01:05.804543 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.204.3:0
--local_ip_for_outbound_sockets=127.0.204.3
--webserver_interface=127.0.204.3
--webserver_port=0
--tserver_master_addrs=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250625 20:01:05.805617 1609 heartbeater.cc:344] Connected to a master server at 127.0.204.61:38005
I20250625 20:01:05.805743 1607 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:05.805881 1607 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:05.805882 1609 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:05.806563 1607 heartbeater.cc:507] Master 127.0.204.60:46191 requested a full tablet report, sending...
I20250625 20:01:05.806563 1609 heartbeater.cc:507] Master 127.0.204.61:38005 requested a full tablet report, sending...
I20250625 20:01:05.807189 1608 heartbeater.cc:344] Connected to a master server at 127.0.204.62:36889
I20250625 20:01:05.807302 1608 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:05.807508 1608 heartbeater.cc:507] Master 127.0.204.62:36889 requested a full tablet report, sending...
I20250625 20:01:05.807646 1233 ts_manager.cc:194] Registered new tserver with Master: c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:05.807945 1162 ts_manager.cc:194] Registered new tserver with Master: c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:05.808169 1091 ts_manager.cc:194] Registered new tserver with Master: c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:05.808405 1233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.204.2:46901
I20250625 20:01:05.869613 1615 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1fcdee0c0aac4013a5e3c77304b474d9)
I20250625 20:01:05.869751 1615 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:05.869869 1616 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1fcdee0c0aac4013a5e3c77304b474d9)
I20250625 20:01:05.869979 1616 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:05.870682 1615 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:05.871044 1172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" is_pre_election: true
I20250625 20:01:05.871147 1172 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5852fb53503b438c92f294e0c44c916d in term 1.
I20250625 20:01:05.871256 1243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" is_pre_election: true
I20250625 20:01:05.871482 1075 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0; no voters:
I20250625 20:01:05.871593 1101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5852fb53503b438c92f294e0c44c916d" is_pre_election: true
I20250625 20:01:05.871702 1101 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f87f9e18e5b948a4a3fb3666f106abd0 in term 1.
I20250625 20:01:05.871869 1615 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250625 20:01:05.871918 1615 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1fcdee0c0aac4013a5e3c77304b474d9)
I20250625 20:01:05.871955 1615 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:01:05.872382 1147 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0; no voters:
I20250625 20:01:05.872753 1616 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:05.872843 1616 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250625 20:01:05.872912 1616 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1fcdee0c0aac4013a5e3c77304b474d9)
I20250625 20:01:05.872961 1616 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:01:05.873071 1243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" is_pre_election: true
I20250625 20:01:05.873920 1615 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:05.873987 1616 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:05.874550 1172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0"
I20250625 20:01:05.874559 1101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5852fb53503b438c92f294e0c44c916d"
I20250625 20:01:05.874684 1172 raft_consensus.cc:2391] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5852fb53503b438c92f294e0c44c916d in current term 2: Already voted for candidate f87f9e18e5b948a4a3fb3666f106abd0 in this term.
I20250625 20:01:05.874698 1101 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f87f9e18e5b948a4a3fb3666f106abd0 in current term 2: Already voted for candidate 5852fb53503b438c92f294e0c44c916d in this term.
I20250625 20:01:05.875349 1615 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 2 election: Requested vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:05.875382 1243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fcdee0c0aac4013a5e3c77304b474d9"
I20250625 20:01:05.875398 1242 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fcdee0c0aac4013a5e3c77304b474d9"
I20250625 20:01:05.875702 1149 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:05.875890 1078 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:05.876111 1616 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 2 election: Requested vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:05.876051 1615 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250625 20:01:05.876276 1616 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250625 20:01:05.890443 1101 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 2 FOLLOWER]: Rejecting Update request from peer 1fcdee0c0aac4013a5e3c77304b474d9 for earlier term 1. Current term is 2. Ops: []
I20250625 20:01:05.890939 1618 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250625 20:01:05.891258 1619 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 1 LEADER]: Stepping down as leader of term 1
I20250625 20:01:05.891321 1619 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1fcdee0c0aac4013a5e3c77304b474d9, State: Running, Role: LEADER
I20250625 20:01:05.891481 1619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:05.891814 1619 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 1 FOLLOWER]: Advancing to term 2
W20250625 20:01:05.917372 1614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:01:05.917627 1614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:05.917699 1614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:01:05.921042 1614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:05.921193 1614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.204.3
I20250625 20:01:05.925277 1614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.204.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.204.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--never_fsync=true
--heap_profile_path=/tmp/kudu.1614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.204.3
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:05.926422 1614 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:05.927579 1614 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:01:05.935426 1625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:05.935441 1627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:05.935524 1614 server_base.cc:1048] running on GCE node
W20250625 20:01:05.935431 1624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:05.936097 1614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:05.936679 1614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:05.937845 1614 hybrid_clock.cc:648] HybridClock initialized: now 1750881665937828 us; error 35 us; skew 500 ppm
I20250625 20:01:05.940310 1614 webserver.cc:469] Webserver started at http://127.0.204.3:33611/ using document root <none> and password file <none>
I20250625 20:01:05.940884 1614 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:05.940943 1614 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:05.941125 1614 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:01:05.942826 1614 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/data/instance:
uuid: "0ed894f2ff26414190d33cd134c36bc3"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
I20250625 20:01:05.943307 1614 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/instance:
uuid: "0ed894f2ff26414190d33cd134c36bc3"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
I20250625 20:01:05.946657 1614 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250625 20:01:05.948840 1633 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:05.949908 1614 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:05.950013 1614 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal
uuid: "0ed894f2ff26414190d33cd134c36bc3"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
I20250625 20:01:05.950102 1614 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:05.970274 1614 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:05.970994 1614 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:05.971167 1614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:05.971791 1614 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:01:05.972896 1614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:01:05.972946 1614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:05.972999 1614 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:01:05.973027 1614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:05.982748 1614 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.3:43255
I20250625 20:01:05.982774 1746 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.3:43255 every 8 connection(s)
I20250625 20:01:05.983825 1614 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/data/info.pb
I20250625 20:01:05.991189 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 1614
I20250625 20:01:05.991305 816 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/instance
I20250625 20:01:05.993253 1748 heartbeater.cc:344] Connected to a master server at 127.0.204.62:36889
I20250625 20:01:05.993480 1749 heartbeater.cc:344] Connected to a master server at 127.0.204.61:38005
I20250625 20:01:05.993717 1747 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:05.993743 1749 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:05.993791 1747 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:05.994431 1747 heartbeater.cc:507] Master 127.0.204.60:46191 requested a full tablet report, sending...
I20250625 20:01:05.994431 1749 heartbeater.cc:507] Master 127.0.204.61:38005 requested a full tablet report, sending...
I20250625 20:01:05.993791 1748 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:05.995155 1748 heartbeater.cc:507] Master 127.0.204.62:36889 requested a full tablet report, sending...
I20250625 20:01:05.995602 1233 ts_manager.cc:194] Registered new tserver with Master: 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:05.995627 1162 ts_manager.cc:194] Registered new tserver with Master: 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:05.995781 1091 ts_manager.cc:194] Registered new tserver with Master: 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:06.009461 816 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 20:01:06.095434 1470 heartbeater.cc:499] Master 127.0.204.60:46191 was elected leader, sending a full tablet report...
I20250625 20:01:06.100495 1291 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 20:01:06.131935 1309 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:06.132014 1309 catalog_manager.cc:1554] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: loading cluster ID for follower catalog manager: success
I20250625 20:01:06.133946 1309 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: acquiring CA information for follower catalog manager: success
I20250625 20:01:06.135118 1309 catalog_manager.cc:1604] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 20:01:06.201357 1327 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:06.201460 1327 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: loading cluster ID for follower catalog manager: success
I20250625 20:01:06.203321 1327 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: acquiring CA information for follower catalog manager: success
I20250625 20:01:06.204550 1327 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 20:01:06.310240 1607 heartbeater.cc:499] Master 127.0.204.60:46191 was elected leader, sending a full tablet report...
I20250625 20:01:06.428565 1761 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:06.428711 1761 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:06.429132 1761 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:06.429572 1242 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" is_pre_election: true
I20250625 20:01:06.429585 1172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" is_pre_election: true
I20250625 20:01:06.429749 1242 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5852fb53503b438c92f294e0c44c916d in term 2.
I20250625 20:01:06.429754 1172 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5852fb53503b438c92f294e0c44c916d in term 2.
I20250625 20:01:06.430068 1078 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d; no voters:
I20250625 20:01:06.430284 1761 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250625 20:01:06.430330 1761 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:06.430357 1761 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 2 FOLLOWER]: Advancing to term 3
I20250625 20:01:06.431228 1761 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:06.431571 1761 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 3 election: Requested vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:06.431896 1242 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1fcdee0c0aac4013a5e3c77304b474d9"
I20250625 20:01:06.432001 1242 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 2 FOLLOWER]: Advancing to term 3
I20250625 20:01:06.432823 1242 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5852fb53503b438c92f294e0c44c916d in term 3.
I20250625 20:01:06.433035 1172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0"
I20250625 20:01:06.433099 1078 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d; no voters:
I20250625 20:01:06.433153 1172 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 2 FOLLOWER]: Advancing to term 3
I20250625 20:01:06.433267 1761 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 3 FOLLOWER]: Leader election won for term 3
I20250625 20:01:06.433468 1761 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 3 LEADER]: Becoming Leader. State: Replica: 5852fb53503b438c92f294e0c44c916d, State: Running, Role: LEADER
I20250625 20:01:06.433770 1761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:06.434005 1172 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5852fb53503b438c92f294e0c44c916d in term 3.
I20250625 20:01:06.435050 1762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5852fb53503b438c92f294e0c44c916d. Latest consensus state: current_term: 3 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:06.435158 1762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:06.435424 1764 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:01:06.436230 1764 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:01:06.436592 1764 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:06.436640 1764 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:01:06.437165 1764 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:01:06.437520 1764 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Loaded TSK: 0
I20250625 20:01:06.438220 1764 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:01:06.438575 1764 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 20:01:06.438875 1764 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250625 20:01:06.457433 816 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu with pid 1130
I20250625 20:01:06.459697 1085 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_9b3ba4829d5b4a07b5b7610472b79b74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:06.460379 1083 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_050aa6ac27914b98b21a7d3b966c893d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:06.459784 1091 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_53d5923f5fec4ca8b2fb669a4cf9395a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:06.459887 1084 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_2e6d0c334cc144fcb1166ef1dc4932f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:06.459985 1086 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_ae9e55cc97654931933bf1eede18454b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:06.459465 1090 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_976af21482a74bc79eb800684cce978e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:06.460332 1089 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_ef511cc05ad8422e98275bbb9b82ecb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:06.460206 1088 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_60554f72b1aa48aa93aaea511a985fe7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:06.460096 1087 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_7b366e40d04d468caf65baf4c5b77004"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:06.460587 1082 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_4c6973f983694e48873d7009b44c2d4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 20:01:06.498311 1749 heartbeater.cc:646] Failed to heartbeat to 127.0.204.61:38005 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
W20250625 20:01:06.505561 1085 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9b3ba4829d5b4a07b5b7610472b79b74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 20:01:06.505561 1083 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_050aa6ac27914b98b21a7d3b966c893d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 20:01:06.505561 1084 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2e6d0c334cc144fcb1166ef1dc4932f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 20:01:06.509253 1086 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ae9e55cc97654931933bf1eede18454b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 20:01:06.509379 1091 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_53d5923f5fec4ca8b2fb669a4cf9395a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 20:01:06.509433 1082 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4c6973f983694e48873d7009b44c2d4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 20:01:06.510072 1087 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7b366e40d04d468caf65baf4c5b77004 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 20:01:06.510670 1088 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_60554f72b1aa48aa93aaea511a985fe7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 20:01:06.511286 1089 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ef511cc05ad8422e98275bbb9b82ecb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 20:01:06.511937 1090 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_976af21482a74bc79eb800684cce978e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 20:01:06.595666 1472 heartbeater.cc:646] Failed to heartbeat to 127.0.204.61:38005 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
W20250625 20:01:06.811326 1609 heartbeater.cc:646] Failed to heartbeat to 127.0.204.61:38005 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:06.937470 1083 catalog_manager.cc:2149] created HMS catalog entry for table [id=82357871d20c400dbb0c5727da020996]
I20250625 20:01:06.940786 1242 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 3 FOLLOWER]: Refusing update from remote peer 5852fb53503b438c92f294e0c44c916d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250625 20:01:06.941771 1801 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250625 20:01:06.947906 1803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5852fb53503b438c92f294e0c44c916d. Latest consensus state: current_term: 3 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:06.948623 1803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:06.947942 1808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:06.948729 1808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:06.949110 1802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:06.949234 1802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: LEADER
W20250625 20:01:06.950234 1075 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d -> Peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Couldn't send request to peer f87f9e18e5b948a4a3fb3666f106abd0. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 20:01:06.952672 1083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.204.3:47507
I20250625 20:01:06.957950 1083 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41858:
table { table_name: "default.Table_050aa6ac27914b98b21a7d3b966c893d" } new_table_name: "default.Table_8302314b76364eadb44f02cc6d58a589" modify_external_catalogs: true
I20250625 20:01:06.968997 1681 tablet_service.cc:1468] Processing CreateTablet for tablet e278e12fe25d48a89a4635ba60e95fac (DEFAULT_TABLE table=default.table_050aa6ac27914b98b21a7d3b966c893d [id=82357871d20c400dbb0c5727da020996]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:06.969930 1681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e278e12fe25d48a89a4635ba60e95fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:06.973927 1541 tablet_service.cc:1468] Processing CreateTablet for tablet e278e12fe25d48a89a4635ba60e95fac (DEFAULT_TABLE table=default.table_050aa6ac27914b98b21a7d3b966c893d [id=82357871d20c400dbb0c5727da020996]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:06.974960 1541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e278e12fe25d48a89a4635ba60e95fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:06.978905 1404 tablet_service.cc:1468] Processing CreateTablet for tablet e278e12fe25d48a89a4635ba60e95fac (DEFAULT_TABLE table=default.table_050aa6ac27914b98b21a7d3b966c893d [id=82357871d20c400dbb0c5727da020996]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:06.979570 1084 catalog_manager.cc:2149] created HMS catalog entry for table [id=c5f1b7b4e1e8405794eea8fc65b77fb0]
I20250625 20:01:06.979897 1404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e278e12fe25d48a89a4635ba60e95fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:06.988361 1817 tablet_bootstrap.cc:492] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:06.988914 1084 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41858:
table { table_name: "default.Table_2e6d0c334cc144fcb1166ef1dc4932f4" } new_table_name: "default.Table_1d0eb88cbdfd4b16bb66bfb3d502672b" modify_external_catalogs: true
I20250625 20:01:06.990671 1817 tablet_bootstrap.cc:654] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:06.991601 1817 log.cc:826] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:06.993367 1817 tablet_bootstrap.cc:492] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:06.993553 1817 ts_tablet_manager.cc:1397] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250625 20:01:06.994501 1540 tablet_service.cc:1468] Processing CreateTablet for tablet 784678767a2849c6a9ae9dc6f977451d (DEFAULT_TABLE table=default.table_2e6d0c334cc144fcb1166ef1dc4932f4 [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:06.994787 1540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 784678767a2849c6a9ae9dc6f977451d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:06.996903 1817 raft_consensus.cc:357] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:06.997129 1817 raft_consensus.cc:383] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:06.997176 1817 raft_consensus.cc:738] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:06.997910 1817 consensus_queue.cc:260] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:06.997977 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.61:38005
--webserver_interface=127.0.204.61
--webserver_port=44189
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 20:01:06.999085 1817 ts_tablet_manager.cc:1428] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250625 20:01:07.000304 1680 tablet_service.cc:1468] Processing CreateTablet for tablet 784678767a2849c6a9ae9dc6f977451d (DEFAULT_TABLE table=default.table_2e6d0c334cc144fcb1166ef1dc4932f4 [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.000633 1680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 784678767a2849c6a9ae9dc6f977451d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.001637 1404 tablet_service.cc:1468] Processing CreateTablet for tablet 784678767a2849c6a9ae9dc6f977451d (DEFAULT_TABLE table=default.table_2e6d0c334cc144fcb1166ef1dc4932f4 [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.001878 1404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 784678767a2849c6a9ae9dc6f977451d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.002120 1471 heartbeater.cc:499] Master 127.0.204.62:36889 was elected leader, sending a full tablet report...
I20250625 20:01:07.003964 1817 tablet_bootstrap.cc:492] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:07.005831 1819 tablet_bootstrap.cc:492] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:07.009093 1817 tablet_bootstrap.cc:654] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.018231 1819 tablet_bootstrap.cc:654] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.018996 1823 tablet_bootstrap.cc:492] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:07.019613 1817 tablet_bootstrap.cc:492] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:07.019742 1817 ts_tablet_manager.cc:1397] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250625 20:01:07.020395 1817 raft_consensus.cc:357] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.020509 1817 raft_consensus.cc:383] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.020540 1817 raft_consensus.cc:738] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.020840 1817 consensus_queue.cc:260] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.021178 1817 ts_tablet_manager.cc:1428] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.028059 1823 tablet_bootstrap.cc:654] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.029309 1819 log.cc:826] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:07.030232 1823 log.cc:826] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:07.033190 1819 tablet_bootstrap.cc:492] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:07.033320 1823 tablet_bootstrap.cc:492] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:07.033367 1819 ts_tablet_manager.cc:1397] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20250625 20:01:07.033492 1823 ts_tablet_manager.cc:1397] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20250625 20:01:07.035966 1819 raft_consensus.cc:357] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.036186 1819 raft_consensus.cc:383] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.036249 1819 raft_consensus.cc:738] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.036665 1819 consensus_queue.cc:260] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.037987 1608 heartbeater.cc:499] Master 127.0.204.62:36889 was elected leader, sending a full tablet report...
I20250625 20:01:07.038578 1819 ts_tablet_manager.cc:1428] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250625 20:01:07.041740 1819 tablet_bootstrap.cc:492] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:07.042428 1823 raft_consensus.cc:357] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:07.042660 1819 tablet_bootstrap.cc:654] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.042657 1823 raft_consensus.cc:383] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.042709 1823 raft_consensus.cc:738] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.043195 1823 consensus_queue.cc:260] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:07.044193 1823 ts_tablet_manager.cc:1428] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250625 20:01:07.044503 1748 heartbeater.cc:499] Master 127.0.204.62:36889 was elected leader, sending a full tablet report...
I20250625 20:01:07.045882 1823 tablet_bootstrap.cc:492] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:07.046902 1823 tablet_bootstrap.cc:654] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
W20250625 20:01:07.047786 1610 tablet.cc:2378] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:01:07.048084 1823 tablet_bootstrap.cc:492] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:07.048168 1823 ts_tablet_manager.cc:1397] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250625 20:01:07.048667 1823 raft_consensus.cc:357] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.048776 1823 raft_consensus.cc:383] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.048806 1823 raft_consensus.cc:738] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.048925 1823 consensus_queue.cc:260] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.049153 1823 ts_tablet_manager.cc:1428] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 20:01:07.050355 1819 tablet_bootstrap.cc:492] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:07.050442 1819 ts_tablet_manager.cc:1397] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250625 20:01:07.050959 1819 raft_consensus.cc:357] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:07.051064 1819 raft_consensus.cc:383] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.051090 1819 raft_consensus.cc:738] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.051232 1819 consensus_queue.cc:260] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:07.051472 1819 ts_tablet_manager.cc:1428] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.059180 1085 catalog_manager.cc:2149] created HMS catalog entry for table [id=4e3fbfc6b591464886cb1e94527676c1]
W20250625 20:01:07.080166 1473 tablet.cc:2378] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:01:07.084103 1681 tablet_service.cc:1468] Processing CreateTablet for tablet 53174bf6005e499c813b813b4a782c19 (DEFAULT_TABLE table=default.table_9b3ba4829d5b4a07b5b7610472b79b74 [id=4e3fbfc6b591464886cb1e94527676c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.084451 1681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53174bf6005e499c813b813b4a782c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.086894 1823 tablet_bootstrap.cc:492] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:07.088001 1823 tablet_bootstrap.cc:654] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.089475 1541 tablet_service.cc:1468] Processing CreateTablet for tablet 53174bf6005e499c813b813b4a782c19 (DEFAULT_TABLE table=default.table_9b3ba4829d5b4a07b5b7610472b79b74 [id=4e3fbfc6b591464886cb1e94527676c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.089797 1541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53174bf6005e499c813b813b4a782c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.091431 1085 master_service.cc:948] ReplaceTablet: received request to replace tablet 784678767a2849c6a9ae9dc6f977451d from {username='slave'} at 127.0.0.1:41858
I20250625 20:01:07.092243 1819 tablet_bootstrap.cc:492] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:07.092414 1404 tablet_service.cc:1468] Processing CreateTablet for tablet 53174bf6005e499c813b813b4a782c19 (DEFAULT_TABLE table=default.table_9b3ba4829d5b4a07b5b7610472b79b74 [id=4e3fbfc6b591464886cb1e94527676c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.092708 1404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53174bf6005e499c813b813b4a782c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.093256 1819 tablet_bootstrap.cc:654] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.095089 1817 tablet_bootstrap.cc:492] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:07.095953 1817 tablet_bootstrap.cc:654] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.097065 1086 catalog_manager.cc:2149] created HMS catalog entry for table [id=e6ca598bb22a45abb67ce2e547f3a809]
I20250625 20:01:07.097589 1817 tablet_bootstrap.cc:492] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:07.097702 1817 ts_tablet_manager.cc:1397] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 20:01:07.098228 1817 raft_consensus.cc:357] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.098317 1817 raft_consensus.cc:383] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.098340 1817 raft_consensus.cc:738] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.098450 1817 consensus_queue.cc:260] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.098754 1817 ts_tablet_manager.cc:1428] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.104646 1086 master_service.cc:948] ReplaceTablet: received request to replace tablet e278e12fe25d48a89a4635ba60e95fac from {username='slave'} at 127.0.0.1:41858
I20250625 20:01:07.104951 1823 tablet_bootstrap.cc:492] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:07.105050 1823 ts_tablet_manager.cc:1397] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.001s
I20250625 20:01:07.105633 1823 raft_consensus.cc:357] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.105765 1823 raft_consensus.cc:383] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.105805 1823 raft_consensus.cc:738] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.105942 1823 consensus_queue.cc:260] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.106242 1823 ts_tablet_manager.cc:1428] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.106885 1819 tablet_bootstrap.cc:492] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:07.106995 1819 ts_tablet_manager.cc:1397] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20250625 20:01:07.107544 1819 raft_consensus.cc:357] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.107658 1819 raft_consensus.cc:383] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.107692 1819 raft_consensus.cc:738] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.107854 1819 consensus_queue.cc:260] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.108152 1819 ts_tablet_manager.cc:1428] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250625 20:01:07.116047 1086 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Sending DeleteTablet for 3 replicas of tablet e278e12fe25d48a89a4635ba60e95fac
I20250625 20:01:07.116510 1086 catalog_manager.cc:6652] ReplaceTablet: tablet e278e12fe25d48a89a4635ba60e95fac deleted and replaced by tablet 6d488e29942c46cb8c67058068ce2b52
I20250625 20:01:07.116873 1085 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Sending DeleteTablet for 3 replicas of tablet 784678767a2849c6a9ae9dc6f977451d
I20250625 20:01:07.117497 1681 tablet_service.cc:1515] Processing DeleteTablet for tablet e278e12fe25d48a89a4635ba60e95fac with delete_type TABLET_DATA_DELETED (replaced by tablet 6d488e29942c46cb8c67058068ce2b52) from {username='slave'} at 127.0.0.1:43114
I20250625 20:01:07.117688 1085 catalog_manager.cc:6652] ReplaceTablet: tablet 784678767a2849c6a9ae9dc6f977451d deleted and replaced by tablet b36de85648bb4236a9320412cdb1cfdf
I20250625 20:01:07.118216 1681 tablet_service.cc:1515] Processing DeleteTablet for tablet 784678767a2849c6a9ae9dc6f977451d with delete_type TABLET_DATA_DELETED (replaced by tablet b36de85648bb4236a9320412cdb1cfdf) from {username='slave'} at 127.0.0.1:43114
I20250625 20:01:07.119364 1541 tablet_service.cc:1515] Processing DeleteTablet for tablet e278e12fe25d48a89a4635ba60e95fac with delete_type TABLET_DATA_DELETED (replaced by tablet 6d488e29942c46cb8c67058068ce2b52) from {username='slave'} at 127.0.0.1:57378
I20250625 20:01:07.119364 1540 tablet_service.cc:1515] Processing DeleteTablet for tablet 784678767a2849c6a9ae9dc6f977451d with delete_type TABLET_DATA_DELETED (replaced by tablet b36de85648bb4236a9320412cdb1cfdf) from {username='slave'} at 127.0.0.1:57378
I20250625 20:01:07.120431 1837 tablet_replica.cc:331] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:07.120940 1404 tablet_service.cc:1515] Processing DeleteTablet for tablet e278e12fe25d48a89a4635ba60e95fac with delete_type TABLET_DATA_DELETED (replaced by tablet 6d488e29942c46cb8c67058068ce2b52) from {username='slave'} at 127.0.0.1:46134
I20250625 20:01:07.121361 1403 tablet_service.cc:1515] Processing DeleteTablet for tablet 784678767a2849c6a9ae9dc6f977451d with delete_type TABLET_DATA_DELETED (replaced by tablet b36de85648bb4236a9320412cdb1cfdf) from {username='slave'} at 127.0.0.1:46134
I20250625 20:01:07.122224 1836 tablet_replica.cc:331] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:07.122648 1836 raft_consensus.cc:2241] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:07.122956 1836 raft_consensus.cc:2270] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:07.124572 1836 ts_tablet_manager.cc:1905] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:07.127365 1836 ts_tablet_manager.cc:1918] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 20:01:07.127514 1836 log.cc:1199] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/e278e12fe25d48a89a4635ba60e95fac
I20250625 20:01:07.127828 1836 ts_tablet_manager.cc:1939] T e278e12fe25d48a89a4635ba60e95fac P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:07.128038 1681 tablet_service.cc:1468] Processing CreateTablet for tablet 3b7836328f054e2c95b12a8f0fe61b2b (DEFAULT_TABLE table=default.table_ae9e55cc97654931933bf1eede18454b [id=e6ca598bb22a45abb67ce2e547f3a809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.128273 1681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7836328f054e2c95b12a8f0fe61b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.129010 1836 tablet_replica.cc:331] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:07.129122 1836 raft_consensus.cc:2241] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:07.129223 1836 raft_consensus.cc:2270] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:07.129946 1829 raft_consensus.cc:491] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:07.130077 1829 raft_consensus.cc:513] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.130242 1823 tablet_bootstrap.cc:492] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:07.130303 1836 ts_tablet_manager.cc:1905] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:07.131276 1829 leader_election.cc:290] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:07.131389 1076 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet e278e12fe25d48a89a4635ba60e95fac (table default.table_050aa6ac27914b98b21a7d3b966c893d [id=82357871d20c400dbb0c5727da020996]) successfully deleted
I20250625 20:01:07.131276 1823 tablet_bootstrap.cc:654] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.132113 1836 ts_tablet_manager.cc:1918] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 20:01:07.132179 1836 log.cc:1199] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/784678767a2849c6a9ae9dc6f977451d
I20250625 20:01:07.132369 1836 ts_tablet_manager.cc:1939] T 784678767a2849c6a9ae9dc6f977451d P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:07.133486 1540 tablet_service.cc:1468] Processing CreateTablet for tablet 3b7836328f054e2c95b12a8f0fe61b2b (DEFAULT_TABLE table=default.table_ae9e55cc97654931933bf1eede18454b [id=e6ca598bb22a45abb67ce2e547f3a809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.133780 1540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7836328f054e2c95b12a8f0fe61b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.135826 1819 tablet_bootstrap.cc:492] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:07.136806 1819 tablet_bootstrap.cc:654] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.138929 1403 tablet_service.cc:1468] Processing CreateTablet for tablet 3b7836328f054e2c95b12a8f0fe61b2b (DEFAULT_TABLE table=default.table_ae9e55cc97654931933bf1eede18454b [id=e6ca598bb22a45abb67ce2e547f3a809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.139077 1837 raft_consensus.cc:2241] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:07.139211 1403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7836328f054e2c95b12a8f0fe61b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.139348 1837 raft_consensus.cc:2270] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:07.139803 1838 tablet_replica.cc:331] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:07.139955 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53174bf6005e499c813b813b4a782c19" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:07.140079 1838 raft_consensus.cc:2241] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:07.140193 1424 raft_consensus.cc:2466] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:07.140290 1838 raft_consensus.cc:2270] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:07.140534 1837 ts_tablet_manager.cc:1905] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:07.140650 1637 leader_election.cc:304] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:07.140923 1829 raft_consensus.cc:2802] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:07.140986 1829 raft_consensus.cc:491] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:07.141005 1819 tablet_bootstrap.cc:492] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:07.141026 1829 raft_consensus.cc:3058] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.141077 1819 ts_tablet_manager.cc:1397] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 20:01:07.141517 1819 raft_consensus.cc:357] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.141611 1819 raft_consensus.cc:383] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.141641 1819 raft_consensus.cc:738] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.141767 1819 consensus_queue.cc:260] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.142014 1819 ts_tablet_manager.cc:1428] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.141973 1829 raft_consensus.cc:513] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.142338 1829 leader_election.cc:290] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:07.143797 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53174bf6005e499c813b813b4a782c19" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:07.143913 1424 raft_consensus.cc:3058] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.144726 1837 ts_tablet_manager.cc:1918] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 20:01:07.144804 1837 log.cc:1199] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/e278e12fe25d48a89a4635ba60e95fac
I20250625 20:01:07.144904 1424 raft_consensus.cc:2466] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:07.145035 1837 ts_tablet_manager.cc:1939] T e278e12fe25d48a89a4635ba60e95fac P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:07.145277 1637 leader_election.cc:304] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:07.145630 1838 ts_tablet_manager.cc:1905] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:07.148304 1817 tablet_bootstrap.cc:492] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:07.148720 1075 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet e278e12fe25d48a89a4635ba60e95fac (table default.table_050aa6ac27914b98b21a7d3b966c893d [id=82357871d20c400dbb0c5727da020996]) successfully deleted
I20250625 20:01:07.148989 1837 tablet_replica.cc:331] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:07.149096 1837 raft_consensus.cc:2241] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:07.149183 1837 raft_consensus.cc:2270] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:07.149252 1817 tablet_bootstrap.cc:654] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.149652 1829 raft_consensus.cc:2802] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:07.149852 1829 raft_consensus.cc:695] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
I20250625 20:01:07.150089 1829 consensus_queue.cc:237] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.150475 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53174bf6005e499c813b813b4a782c19" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:07.150722 1561 raft_consensus.cc:2466] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:07.150997 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53174bf6005e499c813b813b4a782c19" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:07.151093 1560 raft_consensus.cc:3058] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.152143 1560 raft_consensus.cc:2466] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:07.152568 1837 ts_tablet_manager.cc:1905] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:07.154153 1540 tablet_service.cc:1468] Processing CreateTablet for tablet 6d488e29942c46cb8c67058068ce2b52 (DEFAULT_TABLE table=default.table_050aa6ac27914b98b21a7d3b966c893d [id=82357871d20c400dbb0c5727da020996]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.154395 1540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d488e29942c46cb8c67058068ce2b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.154440 1086 master_service.cc:948] ReplaceTablet: received request to replace tablet 784678767a2849c6a9ae9dc6f977451d from {username='slave'} at 127.0.0.1:41858
I20250625 20:01:07.154691 1085 master_service.cc:948] ReplaceTablet: received request to replace tablet 784678767a2849c6a9ae9dc6f977451d from {username='slave'} at 127.0.0.1:41858
I20250625 20:01:07.154738 1086 master_service.cc:948] ReplaceTablet: received request to replace tablet 784678767a2849c6a9ae9dc6f977451d from {username='slave'} at 127.0.0.1:41858
I20250625 20:01:07.155002 1085 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41858:
table { table_name: "default.Table_9b3ba4829d5b4a07b5b7610472b79b74" } new_table_name: "default.Table_d0f8eb84406f4d758e34b162746daa38" modify_external_catalogs: true
I20250625 20:01:07.156194 1541 tablet_service.cc:1468] Processing CreateTablet for tablet b36de85648bb4236a9320412cdb1cfdf (DEFAULT_TABLE table=default.table_2e6d0c334cc144fcb1166ef1dc4932f4 [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.156359 1819 tablet_bootstrap.cc:492] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:07.156440 1541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b36de85648bb4236a9320412cdb1cfdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.157418 1086 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41858:
table { table_name: "default.Table_ae9e55cc97654931933bf1eede18454b" } new_table_name: "default.Table_030bc68e40784cf2a6bd261221a0f738" modify_external_catalogs: true
I20250625 20:01:07.157605 1819 tablet_bootstrap.cc:654] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.148197 1838 ts_tablet_manager.cc:1918] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 20:01:07.158030 1838 log.cc:1199] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/e278e12fe25d48a89a4635ba60e95fac
I20250625 20:01:07.158321 1838 ts_tablet_manager.cc:1939] T e278e12fe25d48a89a4635ba60e95fac P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:07.159471 1681 tablet_service.cc:1468] Processing CreateTablet for tablet 6d488e29942c46cb8c67058068ce2b52 (DEFAULT_TABLE table=default.table_050aa6ac27914b98b21a7d3b966c893d [id=82357871d20c400dbb0c5727da020996]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.159744 1681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d488e29942c46cb8c67058068ce2b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.159794 1078 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet e278e12fe25d48a89a4635ba60e95fac (table default.table_050aa6ac27914b98b21a7d3b966c893d [id=82357871d20c400dbb0c5727da020996]) successfully deleted
I20250625 20:01:07.160005 1838 tablet_replica.cc:331] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:07.160104 1838 raft_consensus.cc:2241] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:07.160192 1838 raft_consensus.cc:2270] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:07.160765 1838 ts_tablet_manager.cc:1905] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:07.161337 1837 ts_tablet_manager.cc:1918] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 20:01:07.161402 1837 log.cc:1199] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/784678767a2849c6a9ae9dc6f977451d
I20250625 20:01:07.161619 1837 ts_tablet_manager.cc:1939] T 784678767a2849c6a9ae9dc6f977451d P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:07.161994 1680 tablet_service.cc:1468] Processing CreateTablet for tablet b36de85648bb4236a9320412cdb1cfdf (DEFAULT_TABLE table=default.table_2e6d0c334cc144fcb1166ef1dc4932f4 [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.162220 1680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b36de85648bb4236a9320412cdb1cfdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.163161 1819 tablet_bootstrap.cc:492] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:07.163240 1819 ts_tablet_manager.cc:1397] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250625 20:01:07.163763 1819 raft_consensus.cc:357] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.163866 1819 raft_consensus.cc:383] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.163897 1819 raft_consensus.cc:738] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.164016 1819 consensus_queue.cc:260] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.164271 1819 ts_tablet_manager.cc:1428] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.164330 1404 tablet_service.cc:1468] Processing CreateTablet for tablet b36de85648bb4236a9320412cdb1cfdf (DEFAULT_TABLE table=default.table_2e6d0c334cc144fcb1166ef1dc4932f4 [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.164453 1819 tablet_bootstrap.cc:492] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:07.164603 1404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b36de85648bb4236a9320412cdb1cfdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.165398 1819 tablet_bootstrap.cc:654] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.167271 1817 tablet_bootstrap.cc:492] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:07.167353 1817 ts_tablet_manager.cc:1397] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20250625 20:01:07.167851 1817 raft_consensus.cc:357] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.167968 1817 raft_consensus.cc:383] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.167999 1817 raft_consensus.cc:738] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.168113 1817 consensus_queue.cc:260] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.168358 1817 ts_tablet_manager.cc:1428] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.168524 1403 tablet_service.cc:1468] Processing CreateTablet for tablet 6d488e29942c46cb8c67058068ce2b52 (DEFAULT_TABLE table=default.table_050aa6ac27914b98b21a7d3b966c893d [id=82357871d20c400dbb0c5727da020996]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.168742 1403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d488e29942c46cb8c67058068ce2b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.170986 1075 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet 784678767a2849c6a9ae9dc6f977451d (table default.table_2e6d0c334cc144fcb1166ef1dc4932f4 [id=c5f1b7b4e1e8405794eea8fc65b77fb0]) successfully deleted
I20250625 20:01:07.171275 1091 catalog_manager.cc:2149] created HMS catalog entry for table [id=224a7f2d5a4d403fa2702fd59ccbf5c9]
I20250625 20:01:07.171815 1838 ts_tablet_manager.cc:1918] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 20:01:07.171882 1838 log.cc:1199] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/784678767a2849c6a9ae9dc6f977451d
I20250625 20:01:07.172071 1838 ts_tablet_manager.cc:1939] T 784678767a2849c6a9ae9dc6f977451d P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:07.172652 1078 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet 784678767a2849c6a9ae9dc6f977451d (table default.table_2e6d0c334cc144fcb1166ef1dc4932f4 [id=c5f1b7b4e1e8405794eea8fc65b77fb0]) successfully deleted
I20250625 20:01:07.173035 1076 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet 784678767a2849c6a9ae9dc6f977451d (table default.table_2e6d0c334cc144fcb1166ef1dc4932f4 [id=c5f1b7b4e1e8405794eea8fc65b77fb0]) successfully deleted
I20250625 20:01:07.168532 1817 tablet_bootstrap.cc:492] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:07.174242 1820 raft_consensus.cc:491] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:07.174294 1817 tablet_bootstrap.cc:654] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.174315 1820 raft_consensus.cc:513] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.175441 1820 leader_election.cc:290] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:07.175459 1823 tablet_bootstrap.cc:492] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:07.175559 1823 ts_tablet_manager.cc:1397] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.000s
I20250625 20:01:07.176074 1823 raft_consensus.cc:357] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.176172 1823 raft_consensus.cc:383] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.176203 1823 raft_consensus.cc:738] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.176302 1823 consensus_queue.cc:260] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.176573 1823 ts_tablet_manager.cc:1428] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 20:01:07.176752 1823 tablet_bootstrap.cc:492] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:07.177670 1823 tablet_bootstrap.cc:654] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.179651 1091 catalog_manager.cc:5582] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3). New cstate: current_term: 1 leader_uuid: "0ed894f2ff26414190d33cd134c36bc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } health_report { overall_health: HEALTHY } } }
I20250625 20:01:07.181963 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b7836328f054e2c95b12a8f0fe61b2b" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:07.182147 1701 raft_consensus.cc:2466] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:07.182626 1358 leader_election.cc:304] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:07.183223 1817 tablet_bootstrap.cc:492] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:07.183315 1817 ts_tablet_manager.cc:1397] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20250625 20:01:07.183627 1820 raft_consensus.cc:2802] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:07.183688 1820 raft_consensus.cc:491] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:07.183770 1820 raft_consensus.cc:3058] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.183844 1817 raft_consensus.cc:357] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.183923 1817 raft_consensus.cc:383] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.183949 1817 raft_consensus.cc:738] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.184054 1817 consensus_queue.cc:260] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.184382 1817 ts_tablet_manager.cc:1428] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.184567 1817 tablet_bootstrap.cc:492] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:07.184701 1820 raft_consensus.cc:513] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.185000 1820 leader_election.cc:290] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:07.185473 1817 tablet_bootstrap.cc:654] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.185832 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b7836328f054e2c95b12a8f0fe61b2b" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:07.185949 1560 raft_consensus.cc:2466] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:07.186142 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b7836328f054e2c95b12a8f0fe61b2b" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:07.186225 1561 raft_consensus.cc:3058] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.186856 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b7836328f054e2c95b12a8f0fe61b2b" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:07.186959 1701 raft_consensus.cc:3058] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.187167 1561 raft_consensus.cc:2466] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:07.187959 1701 raft_consensus.cc:2466] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:07.188238 1358 leader_election.cc:304] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:07.188457 1820 raft_consensus.cc:2802] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:07.188546 1820 raft_consensus.cc:695] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:07.188771 1820 consensus_queue.cc:237] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.191303 1823 tablet_bootstrap.cc:492] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:07.191398 1823 ts_tablet_manager.cc:1397] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20250625 20:01:07.191900 1823 raft_consensus.cc:357] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.191995 1823 raft_consensus.cc:383] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.192023 1823 raft_consensus.cc:738] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.192157 1823 consensus_queue.cc:260] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.192440 1823 ts_tablet_manager.cc:1428] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.192642 1823 tablet_bootstrap.cc:492] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:07.193315 1819 tablet_bootstrap.cc:492] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:07.193398 1819 ts_tablet_manager.cc:1397] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20250625 20:01:07.193660 1823 tablet_bootstrap.cc:654] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.193877 1819 raft_consensus.cc:357] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.193977 1819 raft_consensus.cc:383] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.194008 1819 raft_consensus.cc:738] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
W20250625 20:01:07.197168 1821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:01:07.197489 1821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:07.197564 1821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250625 20:01:07.197736 1817 tablet_bootstrap.cc:492] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:07.197832 1817 ts_tablet_manager.cc:1397] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250625 20:01:07.198326 1817 raft_consensus.cc:357] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.198426 1817 raft_consensus.cc:383] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.198457 1817 raft_consensus.cc:738] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.198585 1817 consensus_queue.cc:260] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.198845 1817 ts_tablet_manager.cc:1428] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.202301 1680 tablet_service.cc:1468] Processing CreateTablet for tablet 7c78cfa2f9504af6910dabc002fbe1da (DEFAULT_TABLE table=default.table_53d5923f5fec4ca8b2fb669a4cf9395a [id=224a7f2d5a4d403fa2702fd59ccbf5c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.202301 1541 tablet_service.cc:1468] Processing CreateTablet for tablet 7c78cfa2f9504af6910dabc002fbe1da (DEFAULT_TABLE table=default.table_53d5923f5fec4ca8b2fb669a4cf9395a [id=224a7f2d5a4d403fa2702fd59ccbf5c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.202585 1541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c78cfa2f9504af6910dabc002fbe1da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.202585 1680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c78cfa2f9504af6910dabc002fbe1da. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 20:01:07.202796 1821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:01:07.202858 1821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:07.202888 1821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:01:07.202914 1821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:01:07.205312 1403 tablet_service.cc:1468] Processing CreateTablet for tablet 7c78cfa2f9504af6910dabc002fbe1da (DEFAULT_TABLE table=default.table_53d5923f5fec4ca8b2fb669a4cf9395a [id=224a7f2d5a4d403fa2702fd59ccbf5c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.205574 1403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c78cfa2f9504af6910dabc002fbe1da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.207885 1817 tablet_bootstrap.cc:492] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:07.208616 1821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.61:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.204.61
--webserver_port=44189
--never_fsync=true
--heap_profile_path=/tmp/kudu.1821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:07.208863 1817 tablet_bootstrap.cc:654] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.210111 1821 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:07.210568 1091 catalog_manager.cc:5582] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1). New cstate: current_term: 1 leader_uuid: "27aa5b53b16747f9a72485aaf2417b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } health_report { overall_health: UNKNOWN } } }
I20250625 20:01:07.211750 1821 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250625 20:01:07.211513 1819 consensus_queue.cc:260] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.212824 1823 tablet_bootstrap.cc:492] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:07.212920 1823 ts_tablet_manager.cc:1397] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20250625 20:01:07.213442 1823 raft_consensus.cc:357] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.213560 1823 raft_consensus.cc:383] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.213595 1823 raft_consensus.cc:738] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.213711 1823 consensus_queue.cc:260] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.213990 1823 ts_tablet_manager.cc:1428] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.214172 1823 tablet_bootstrap.cc:492] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:07.215121 1823 tablet_bootstrap.cc:654] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.218923 1819 ts_tablet_manager.cc:1428] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.025s user 0.001s sys 0.000s
I20250625 20:01:07.220748 1817 tablet_bootstrap.cc:492] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:07.220850 1817 ts_tablet_manager.cc:1397] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250625 20:01:07.220882 1819 tablet_bootstrap.cc:492] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:07.221315 1817 raft_consensus.cc:357] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.221432 1817 raft_consensus.cc:383] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.221472 1817 raft_consensus.cc:738] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
W20250625 20:01:07.221436 1861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:07.221611 1817 consensus_queue.cc:260] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
W20250625 20:01:07.221788 1863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:07.221891 1817 ts_tablet_manager.cc:1428] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20250625 20:01:07.222354 1860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:07.222412 1819 tablet_bootstrap.cc:654] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.222687 1821 server_base.cc:1048] running on GCE node
I20250625 20:01:07.223374 1821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:07.224655 1821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:07.224915 1819 tablet_bootstrap.cc:492] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:07.225033 1819 ts_tablet_manager.cc:1397] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250625 20:01:07.225469 1823 tablet_bootstrap.cc:492] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:07.225584 1823 ts_tablet_manager.cc:1397] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20250625 20:01:07.225695 1819 raft_consensus.cc:357] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.225801 1819 raft_consensus.cc:383] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.225839 1819 raft_consensus.cc:738] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.225963 1819 consensus_queue.cc:260] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.226043 1823 raft_consensus.cc:357] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.226156 1823 raft_consensus.cc:383] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.226194 1823 raft_consensus.cc:738] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.226254 1819 ts_tablet_manager.cc:1428] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 20:01:07.226305 1823 consensus_queue.cc:260] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.226583 1823 ts_tablet_manager.cc:1428] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.231840 1091 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_fd28d4c5b48245c382a334b98a3066df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:07.231990 1821 hybrid_clock.cc:648] HybridClock initialized: now 1750881667231232 us; error 766 us; skew 500 ppm
I20250625 20:01:07.234562 1821 webserver.cc:469] Webserver started at http://127.0.204.61:44189/ using document root <none> and password file <none>
I20250625 20:01:07.235311 1821 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:07.235416 1821 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:07.242476 1821 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.002s
I20250625 20:01:07.244724 1870 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:07.245931 1821 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250625 20:01:07.246038 1821 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
uuid: "f87f9e18e5b948a4a3fb3666f106abd0"
format_stamp: "Formatted at 2025-06-25 20:01:04 on dist-test-slave-lc57"
I20250625 20:01:07.246418 1821 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:07.255278 1082 catalog_manager.cc:2149] created HMS catalog entry for table [id=af2791dc33fb464db4a937aebe316482]
W20250625 20:01:07.255446 1750 tablet.cc:2378] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:01:07.260437 1082 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41858:
table { table_name: "default.Table_53d5923f5fec4ca8b2fb669a4cf9395a" } modify_external_catalogs: true
I20250625 20:01:07.260619 1082 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41858:
table { table_name: "default.Table_53d5923f5fec4ca8b2fb669a4cf9395a" } modify_external_catalogs: true
I20250625 20:01:07.265249 1821 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:07.266016 1821 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:07.266182 1821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:07.266158 1541 tablet_service.cc:1468] Processing CreateTablet for tablet 936500cf83f84a1197231f4de02cd55b (DEFAULT_TABLE table=default.table_4c6973f983694e48873d7009b44c2d4d [id=af2791dc33fb464db4a937aebe316482]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.266256 1680 tablet_service.cc:1468] Processing CreateTablet for tablet 936500cf83f84a1197231f4de02cd55b (DEFAULT_TABLE table=default.table_4c6973f983694e48873d7009b44c2d4d [id=af2791dc33fb464db4a937aebe316482]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.266448 1541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 936500cf83f84a1197231f4de02cd55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.266522 1680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 936500cf83f84a1197231f4de02cd55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.268816 1819 tablet_bootstrap.cc:492] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:07.269063 1823 tablet_bootstrap.cc:492] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:07.269914 1819 tablet_bootstrap.cc:654] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.270367 1823 tablet_bootstrap.cc:654] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.272152 1403 tablet_service.cc:1468] Processing CreateTablet for tablet 936500cf83f84a1197231f4de02cd55b (DEFAULT_TABLE table=default.table_4c6973f983694e48873d7009b44c2d4d [id=af2791dc33fb464db4a937aebe316482]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.272428 1403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 936500cf83f84a1197231f4de02cd55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.274667 1817 tablet_bootstrap.cc:492] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:07.275053 1821 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.61:38005
I20250625 20:01:07.275060 1924 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.61:38005 every 8 connection(s)
I20250625 20:01:07.275888 1817 tablet_bootstrap.cc:654] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.276671 1823 tablet_bootstrap.cc:492] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:07.276769 1823 ts_tablet_manager.cc:1397] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250625 20:01:07.277256 1823 raft_consensus.cc:357] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.277380 1823 raft_consensus.cc:383] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.277412 1823 raft_consensus.cc:738] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.277557 1823 consensus_queue.cc:260] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.278450 1823 ts_tablet_manager.cc:1428] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:07.279738 1821 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/info.pb
I20250625 20:01:07.280896 1819 tablet_bootstrap.cc:492] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:07.280910 1817 tablet_bootstrap.cc:492] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:07.280987 1819 ts_tablet_manager.cc:1397] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20250625 20:01:07.280995 1817 ts_tablet_manager.cc:1397] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250625 20:01:07.281392 1819 raft_consensus.cc:357] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.281482 1819 raft_consensus.cc:383] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.281520 1819 raft_consensus.cc:738] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.281563 1850 raft_consensus.cc:491] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:07.281486 1817 raft_consensus.cc:357] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.281631 1817 raft_consensus.cc:383] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.281677 1817 raft_consensus.cc:738] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.281652 1819 consensus_queue.cc:260] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.281780 1817 consensus_queue.cc:260] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.281899 1819 ts_tablet_manager.cc:1428] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.282065 1817 ts_tablet_manager.cc:1428] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.281633 1850 raft_consensus.cc:513] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.284648 1850 leader_election.cc:290] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:07.285316 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c78cfa2f9504af6910dabc002fbe1da" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:07.285449 1561 raft_consensus.cc:2466] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:07.286022 1357 leader_election.cc:304] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:07.286463 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c78cfa2f9504af6910dabc002fbe1da" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:07.286619 1701 raft_consensus.cc:2466] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:07.287845 1850 raft_consensus.cc:2802] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:07.287902 1850 raft_consensus.cc:491] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:07.287935 1850 raft_consensus.cc:3058] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.288971 1850 raft_consensus.cc:513] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.289327 1850 leader_election.cc:290] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:07.289415 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 1821
I20250625 20:01:07.289716 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c78cfa2f9504af6910dabc002fbe1da" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:07.289829 1561 raft_consensus.cc:3058] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.290772 1561 raft_consensus.cc:2466] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:07.291126 1357 leader_election.cc:304] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:07.291391 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c78cfa2f9504af6910dabc002fbe1da" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:07.291507 1701 raft_consensus.cc:3058] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.293211 1701 raft_consensus.cc:2466] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:07.293994 1850 raft_consensus.cc:2802] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:07.294085 1850 raft_consensus.cc:695] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:07.294217 1850 consensus_queue.cc:237] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:07.295212 1827 raft_consensus.cc:491] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:07.295246 1926 sys_catalog.cc:263] Verifying existing consensus state
I20250625 20:01:07.295300 1827 raft_consensus.cc:513] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.296468 1827 leader_election.cc:290] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:07.297474 1926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Bootstrap starting.
I20250625 20:01:07.302489 1087 catalog_manager.cc:2149] created HMS catalog entry for table [id=18d82881922e465cbcc90e87dfe4a5e1]
I20250625 20:01:07.304473 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d488e29942c46cb8c67058068ce2b52" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:07.304608 1424 raft_consensus.cc:2466] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:07.305011 1497 leader_election.cc:304] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:07.305300 1827 raft_consensus.cc:2802] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:07.305357 1827 raft_consensus.cc:491] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:07.305393 1827 raft_consensus.cc:3058] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.306416 1827 raft_consensus.cc:513] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.306820 1827 leader_election.cc:290] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:07.307548 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d488e29942c46cb8c67058068ce2b52" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:07.307674 1424 raft_consensus.cc:3058] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.308687 1424 raft_consensus.cc:2466] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:07.309041 1497 leader_election.cc:304] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:07.309264 1827 raft_consensus.cc:2802] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:07.309465 1827 raft_consensus.cc:695] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Becoming Leader. State: Replica: c73846c0665d4d53a274d053946107df, State: Running, Role: LEADER
I20250625 20:01:07.309716 1827 consensus_queue.cc:237] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.314623 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d488e29942c46cb8c67058068ce2b52" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:07.314772 1701 raft_consensus.cc:2466] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:07.315004 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d488e29942c46cb8c67058068ce2b52" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:07.315092 1700 raft_consensus.cc:3058] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.317248 1700 raft_consensus.cc:2466] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:07.318540 1680 tablet_service.cc:1468] Processing CreateTablet for tablet 9f41fcc4d1a14c3c94a3f20115926eec (DEFAULT_TABLE table=default.table_7b366e40d04d468caf65baf4c5b77004 [id=18d82881922e465cbcc90e87dfe4a5e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.318868 1680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f41fcc4d1a14c3c94a3f20115926eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.319823 1087 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_a31a093a024346aab31bc8d55a00253d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:07.320621 1541 tablet_service.cc:1468] Processing CreateTablet for tablet 9f41fcc4d1a14c3c94a3f20115926eec (DEFAULT_TABLE table=default.table_7b366e40d04d468caf65baf4c5b77004 [id=18d82881922e465cbcc90e87dfe4a5e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.320876 1541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f41fcc4d1a14c3c94a3f20115926eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.321991 1403 tablet_service.cc:1468] Processing CreateTablet for tablet 9f41fcc4d1a14c3c94a3f20115926eec (DEFAULT_TABLE table=default.table_7b366e40d04d468caf65baf4c5b77004 [id=18d82881922e465cbcc90e87dfe4a5e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.322250 1403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f41fcc4d1a14c3c94a3f20115926eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.324721 1819 tablet_bootstrap.cc:492] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:07.325495 1926 log.cc:826] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:07.325603 1817 tablet_bootstrap.cc:492] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:07.325816 1819 tablet_bootstrap.cc:654] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.326601 1817 tablet_bootstrap.cc:654] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.327891 1819 tablet_bootstrap.cc:492] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:07.327976 1819 ts_tablet_manager.cc:1397] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 20:01:07.328445 1819 raft_consensus.cc:357] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.328574 1819 raft_consensus.cc:383] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.328605 1819 raft_consensus.cc:738] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.328716 1819 consensus_queue.cc:260] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.328951 1819 ts_tablet_manager.cc:1428] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.330447 1823 tablet_bootstrap.cc:492] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:07.331470 1823 tablet_bootstrap.cc:654] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.337239 1609 heartbeater.cc:344] Connected to a master server at 127.0.204.61:38005
I20250625 20:01:07.337819 1926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:01:07.338183 1926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Bootstrap complete.
I20250625 20:01:07.341709 1926 raft_consensus.cc:357] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:07.342135 1926 raft_consensus.cc:738] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87f9e18e5b948a4a3fb3666f106abd0, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.342978 1926 consensus_queue.cc:260] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:07.344286 1926 sys_catalog.cc:564] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:01:07.346385 1472 heartbeater.cc:344] Connected to a master server at 127.0.204.61:38005
I20250625 20:01:07.349939 1888 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" instance_seqno: 1750881665574847) as {username='slave'} at 127.0.204.1:40007; Asking this server to re-register.
I20250625 20:01:07.351826 1472 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:07.353320 1888 ts_manager.cc:194] Registered new tserver with Master: 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:07.355901 1941 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:07.356015 1941 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:07.357409 1749 heartbeater.cc:344] Connected to a master server at 127.0.204.61:38005
I20250625 20:01:07.359138 1888 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" instance_seqno: 1750881665979916) as {username='slave'} at 127.0.204.3:52925; Asking this server to re-register.
I20250625 20:01:07.359629 1749 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:07.360508 1817 tablet_bootstrap.cc:492] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:07.360620 1823 tablet_bootstrap.cc:492] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:07.360625 1817 ts_tablet_manager.cc:1397] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20250625 20:01:07.360705 1823 ts_tablet_manager.cc:1397] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20250625 20:01:07.360740 1888 ts_manager.cc:194] Registered new tserver with Master: 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:07.361158 1817 raft_consensus.cc:357] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.361258 1817 raft_consensus.cc:383] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.361191 1823 raft_consensus.cc:357] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.361289 1817 raft_consensus.cc:738] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.361297 1823 raft_consensus.cc:383] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.361327 1823 raft_consensus.cc:738] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.361415 1817 consensus_queue.cc:260] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.361441 1823 consensus_queue.cc:260] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.361696 1817 ts_tablet_manager.cc:1428] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.361716 1823 ts_tablet_manager.cc:1428] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.366225 1088 catalog_manager.cc:2149] created HMS catalog entry for table [id=e46f78618ca648e19c3935d0bf874a35]
I20250625 20:01:07.372053 1955 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:07.372130 1955 catalog_manager.cc:1554] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: loading cluster ID for follower catalog manager: success
I20250625 20:01:07.375102 1955 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: acquiring CA information for follower catalog manager: success
I20250625 20:01:07.375197 1088 catalog_manager.cc:5582] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1). New cstate: current_term: 1 leader_uuid: "27aa5b53b16747f9a72485aaf2417b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } health_report { overall_health: UNKNOWN } } }
I20250625 20:01:07.376235 1680 tablet_service.cc:1468] Processing CreateTablet for tablet 44500be7b904460fa32515082b47a6f0 (DEFAULT_TABLE table=default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.376524 1680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44500be7b904460fa32515082b47a6f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.376576 1955 catalog_manager.cc:1604] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 20:01:07.378945 1541 tablet_service.cc:1468] Processing CreateTablet for tablet 44500be7b904460fa32515082b47a6f0 (DEFAULT_TABLE table=default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.379225 1541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44500be7b904460fa32515082b47a6f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.381510 1823 tablet_bootstrap.cc:492] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:07.382725 1823 tablet_bootstrap.cc:654] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.383409 1819 tablet_bootstrap.cc:492] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:07.384380 1819 tablet_bootstrap.cc:654] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.386600 1403 tablet_service.cc:1468] Processing CreateTablet for tablet 44500be7b904460fa32515082b47a6f0 (DEFAULT_TABLE table=default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.386879 1403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44500be7b904460fa32515082b47a6f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.389045 1817 tablet_bootstrap.cc:492] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:07.390154 1817 tablet_bootstrap.cc:654] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.394788 1088 catalog_manager.cc:5582] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df reported cstate change: term changed from 0 to 1, leader changed from <none> to c73846c0665d4d53a274d053946107df (127.0.204.2). New cstate: current_term: 1 leader_uuid: "c73846c0665d4d53a274d053946107df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } health_report { overall_health: UNKNOWN } } }
I20250625 20:01:07.401363 1823 tablet_bootstrap.cc:492] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:07.401470 1823 ts_tablet_manager.cc:1397] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20250625 20:01:07.401960 1823 raft_consensus.cc:357] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.402069 1823 raft_consensus.cc:383] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.402098 1823 raft_consensus.cc:738] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.405750 1823 consensus_queue.cc:260] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.408879 1823 ts_tablet_manager.cc:1428] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250625 20:01:07.416172 1088 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41858:
table { table_name: "default.Table_4c6973f983694e48873d7009b44c2d4d" } modify_external_catalogs: true
I20250625 20:01:07.416402 1088 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41858:
table { table_name: "default.Table_4c6973f983694e48873d7009b44c2d4d" } modify_external_catalogs: true
I20250625 20:01:07.417284 1819 tablet_bootstrap.cc:492] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:07.417387 1819 ts_tablet_manager.cc:1397] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20250625 20:01:07.417918 1819 raft_consensus.cc:357] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.418020 1819 raft_consensus.cc:383] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.418048 1819 raft_consensus.cc:738] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.418176 1819 consensus_queue.cc:260] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.418440 1819 ts_tablet_manager.cc:1428] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.420859 1888 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c73846c0665d4d53a274d053946107df" instance_seqno: 1750881665791183) as {username='slave'} at 127.0.204.2:40645; Asking this server to re-register.
I20250625 20:01:07.422346 1609 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:07.423393 1888 ts_manager.cc:194] Registered new tserver with Master: c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:07.424680 1817 tablet_bootstrap.cc:492] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:07.424777 1817 ts_tablet_manager.cc:1397] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20250625 20:01:07.425267 1817 raft_consensus.cc:357] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.425362 1817 raft_consensus.cc:383] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.425390 1817 raft_consensus.cc:738] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.425505 1817 consensus_queue.cc:260] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.425782 1817 ts_tablet_manager.cc:1428] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.426096 1850 raft_consensus.cc:491] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:07.426169 1850 raft_consensus.cc:513] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.426532 1850 leader_election.cc:290] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:07.427760 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b36de85648bb4236a9320412cdb1cfdf" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:07.427884 1561 raft_consensus.cc:2466] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:07.428884 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b36de85648bb4236a9320412cdb1cfdf" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:07.429013 1700 raft_consensus.cc:2466] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:07.430135 1357 leader_election.cc:304] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:07.430301 1820 raft_consensus.cc:2802] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:07.430341 1820 raft_consensus.cc:491] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:07.430370 1820 raft_consensus.cc:3058] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.431368 1820 raft_consensus.cc:513] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.431763 1820 leader_election.cc:290] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:07.433466 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b36de85648bb4236a9320412cdb1cfdf" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:07.433612 1700 raft_consensus.cc:3058] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.434645 1700 raft_consensus.cc:2466] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:07.435151 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b36de85648bb4236a9320412cdb1cfdf" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:07.435202 1358 leader_election.cc:304] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:07.435271 1561 raft_consensus.cc:3058] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.435381 1820 raft_consensus.cc:2802] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:07.435472 1820 raft_consensus.cc:695] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:07.435638 1820 consensus_queue.cc:237] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.436221 1561 raft_consensus.cc:2466] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:07.441668 1089 catalog_manager.cc:2149] created HMS catalog entry for table [id=5483979a624546ea93cff095a5cd2a36]
I20250625 20:01:07.445887 1843 raft_consensus.cc:491] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:07.446010 1843 raft_consensus.cc:513] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.446417 1843 leader_election.cc:290] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:07.447641 1089 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_3629c083428e4ae0a977ffb16af83bc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:07.451726 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "936500cf83f84a1197231f4de02cd55b" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:07.451885 1424 raft_consensus.cc:2466] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:07.452306 1637 leader_election.cc:304] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:07.452502 1843 raft_consensus.cc:2802] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:07.452564 1843 raft_consensus.cc:491] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:07.452621 1843 raft_consensus.cc:3058] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.453645 1843 raft_consensus.cc:513] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.454071 1843 leader_election.cc:290] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:07.454445 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "936500cf83f84a1197231f4de02cd55b" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:07.454535 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "936500cf83f84a1197231f4de02cd55b" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:07.454581 1424 raft_consensus.cc:3058] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.454660 1560 raft_consensus.cc:3058] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.455561 1560 raft_consensus.cc:2466] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:07.455585 1424 raft_consensus.cc:2466] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:07.455991 1634 leader_election.cc:304] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:07.456247 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "936500cf83f84a1197231f4de02cd55b" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:07.456377 1561 raft_consensus.cc:2374] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1. Re-sending same reply.
I20250625 20:01:07.458832 1680 tablet_service.cc:1468] Processing CreateTablet for tablet 1c5e0c439cdb4e67a155bd125982f971 (DEFAULT_TABLE table=default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.459115 1680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5e0c439cdb4e67a155bd125982f971. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.460271 1541 tablet_service.cc:1468] Processing CreateTablet for tablet 1c5e0c439cdb4e67a155bd125982f971 (DEFAULT_TABLE table=default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.460541 1541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5e0c439cdb4e67a155bd125982f971. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.462569 1819 tablet_bootstrap.cc:492] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:07.463605 1819 tablet_bootstrap.cc:654] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.464660 1403 tablet_service.cc:1468] Processing CreateTablet for tablet 1c5e0c439cdb4e67a155bd125982f971 (DEFAULT_TABLE table=default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.464979 1403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5e0c439cdb4e67a155bd125982f971. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.466087 1819 tablet_bootstrap.cc:492] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:07.466161 1819 ts_tablet_manager.cc:1397] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250625 20:01:07.466706 1819 raft_consensus.cc:357] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.466804 1819 raft_consensus.cc:383] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.466832 1819 raft_consensus.cc:738] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.466959 1819 consensus_queue.cc:260] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.467209 1819 ts_tablet_manager.cc:1428] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.467335 1817 tablet_bootstrap.cc:492] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:07.468377 1817 tablet_bootstrap.cc:654] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.471858 1843 raft_consensus.cc:2802] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:07.471967 1843 raft_consensus.cc:695] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
I20250625 20:01:07.472265 1823 tablet_bootstrap.cc:492] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:07.473333 1823 tablet_bootstrap.cc:654] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.472095 1843 consensus_queue.cc:237] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.478478 1823 tablet_bootstrap.cc:492] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:07.478591 1823 ts_tablet_manager.cc:1397] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250625 20:01:07.479082 1823 raft_consensus.cc:357] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.479183 1823 raft_consensus.cc:383] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.479213 1823 raft_consensus.cc:738] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.479339 1823 consensus_queue.cc:260] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.479622 1823 ts_tablet_manager.cc:1428] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.479764 1090 catalog_manager.cc:2149] created HMS catalog entry for table [id=a8f816328d6c43b8b796227333a1d583]
I20250625 20:01:07.481784 1817 tablet_bootstrap.cc:492] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:07.481892 1817 ts_tablet_manager.cc:1397] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20250625 20:01:07.482369 1817 raft_consensus.cc:357] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.482483 1817 raft_consensus.cc:383] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.482522 1817 raft_consensus.cc:738] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.482658 1817 consensus_queue.cc:260] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.482915 1817 ts_tablet_manager.cc:1428] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:07.490147 1090 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41858 (request call id 27) took 1031 ms. Trace:
I20250625 20:01:07.490252 1090 rpcz_store.cc:276] 0625 20:01:06.459123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:06.459192 (+ 69us) service_pool.cc:225] Handling call
0625 20:01:06.512017 (+ 52825us) catalog_manager.cc:2076] Acquired catalog manager lock
0625 20:01:06.512284 (+ 267us) catalog_manager.cc:2122] Created new table and tablet info
0625 20:01:07.479755 (+967471us) catalog_manager.cc:2148] Created new table in HMS catalog
0625 20:01:07.489940 (+ 10185us) catalog_manager.cc:2182] Wrote table and tablets to system table
0625 20:01:07.490036 (+ 96us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0625 20:01:07.490126 (+ 90us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 20:01:07.480250 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 20:01:07.480352 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0625 20:01:07.480357 (+ 5us) write_op.cc:435] Acquired schema lock
0625 20:01:07.480358 (+ 1us) tablet.cc:660] Decoding operations
0625 20:01:07.480450 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0625 20:01:07.480461 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0625 20:01:07.480464 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0625 20:01:07.480519 (+ 55us) tablet.cc:699] Row locks acquired
0625 20:01:07.480520 (+ 1us) write_op.cc:260] PREPARE: finished
0625 20:01:07.480549 (+ 29us) write_op.cc:270] Start()
0625 20:01:07.480586 (+ 37us) write_op.cc:276] Timestamp: P: 1750881667480476 usec, L: 0
0625 20:01:07.480588 (+ 2us) op_driver.cc:348] REPLICATION: starting
0625 20:01:07.480722 (+ 134us) log.cc:844] Serialized 432 byte log entry
0625 20:01:07.489414 (+ 8692us) op_driver.cc:464] REPLICATION: finished
0625 20:01:07.489545 (+ 131us) write_op.cc:301] APPLY: starting
0625 20:01:07.489564 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0625 20:01:07.489596 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0625 20:01:07.489611 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 20:01:07.489689 (+ 78us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 20:01:07.489718 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0625 20:01:07.489723 (+ 5us) write_op.cc:312] APPLY: finished
0625 20:01:07.489790 (+ 67us) log.cc:844] Serialized 28 byte log entry
0625 20:01:07.489840 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0625 20:01:07.489859 (+ 19us) write_op.cc:454] Released schema lock
0625 20:01:07.489864 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0625 20:01:07.489932 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0625 20:01:07.489934 (+ 2us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":929271,"Hive Metastore.run_cpu_time_us":198,"Hive Metastore.run_wall_time_us":38074,"mutex_wait_us":67,"child_traces":[["op",{"apply.queue_time_us":73,"apply.run_cpu_time_us":388,"apply.run_wall_time_us":387,"num_ops":2,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":558,"prepare.run_wall_time_us":558,"raft.queue_time_us":281,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":160,"replication_time_us":8602}]]}
I20250625 20:01:07.491631 1090 catalog_manager.cc:5582] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1). New cstate: current_term: 1 leader_uuid: "27aa5b53b16747f9a72485aaf2417b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } health_report { overall_health: UNKNOWN } } }
I20250625 20:01:07.501672 1680 tablet_service.cc:1468] Processing CreateTablet for tablet bb6c755fce6640d2a0b13a449d3adfb9 (DEFAULT_TABLE table=default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.502012 1680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb6c755fce6640d2a0b13a449d3adfb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.501958 1090 catalog_manager.cc:5582] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3). New cstate: current_term: 1 leader_uuid: "0ed894f2ff26414190d33cd134c36bc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } health_report { overall_health: HEALTHY } } }
I20250625 20:01:07.502908 1541 tablet_service.cc:1468] Processing CreateTablet for tablet bb6c755fce6640d2a0b13a449d3adfb9 (DEFAULT_TABLE table=default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.503160 1541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb6c755fce6640d2a0b13a449d3adfb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.505051 1403 tablet_service.cc:1468] Processing CreateTablet for tablet bb6c755fce6640d2a0b13a449d3adfb9 (DEFAULT_TABLE table=default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:07.505219 1819 tablet_bootstrap.cc:492] T bb6c755fce6640d2a0b13a449d3adfb9 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:07.505326 1403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb6c755fce6640d2a0b13a449d3adfb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:07.506172 1819 tablet_bootstrap.cc:654] T bb6c755fce6640d2a0b13a449d3adfb9 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.508301 1823 tablet_bootstrap.cc:492] T bb6c755fce6640d2a0b13a449d3adfb9 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:07.508826 1843 raft_consensus.cc:491] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:07.508919 1843 raft_consensus.cc:513] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.509292 1843 leader_election.cc:290] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:07.509410 1817 tablet_bootstrap.cc:492] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:07.509676 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44500be7b904460fa32515082b47a6f0" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:07.509790 1561 raft_consensus.cc:2466] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:07.510206 1634 leader_election.cc:304] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:07.510576 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44500be7b904460fa32515082b47a6f0" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:07.510707 1424 raft_consensus.cc:2466] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:07.511328 1843 raft_consensus.cc:2802] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:07.511384 1843 raft_consensus.cc:491] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:07.511420 1843 raft_consensus.cc:3058] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.511492 1823 tablet_bootstrap.cc:654] T bb6c755fce6640d2a0b13a449d3adfb9 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.512374 1843 raft_consensus.cc:513] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.512729 1843 leader_election.cc:290] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:07.513131 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44500be7b904460fa32515082b47a6f0" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:07.513231 1561 raft_consensus.cc:3058] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.513456 1817 tablet_bootstrap.cc:654] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:07.514092 1561 raft_consensus.cc:2466] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:07.514493 1634 leader_election.cc:304] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:07.514850 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44500be7b904460fa32515082b47a6f0" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:07.514955 1424 raft_consensus.cc:3058] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.515802 1424 raft_consensus.cc:2466] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:07.517976 1823 tablet_bootstrap.cc:492] T bb6c755fce6640d2a0b13a449d3adfb9 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:07.518066 1823 ts_tablet_manager.cc:1397] T bb6c755fce6640d2a0b13a449d3adfb9 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20250625 20:01:07.518301 1090 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41858:
table { table_name: "default.Table_7b366e40d04d468caf65baf4c5b77004" } modify_external_catalogs: true
I20250625 20:01:07.518420 1090 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41858:
table { table_name: "default.Table_7b366e40d04d468caf65baf4c5b77004" } modify_external_catalogs: true
I20250625 20:01:07.518515 1823 raft_consensus.cc:357] T bb6c755fce6640d2a0b13a449d3adfb9 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.518610 1823 raft_consensus.cc:383] T bb6c755fce6640d2a0b13a449d3adfb9 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.518639 1823 raft_consensus.cc:738] T bb6c755fce6640d2a0b13a449d3adfb9 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.518755 1823 consensus_queue.cc:260] T bb6c755fce6640d2a0b13a449d3adfb9 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.519019 1823 ts_tablet_manager.cc:1428] T bb6c755fce6640d2a0b13a449d3adfb9 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 20:01:07.519163 1829 raft_consensus.cc:2802] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:07.519310 1829 raft_consensus.cc:695] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
I20250625 20:01:07.519426 1829 consensus_queue.cc:237] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.519500 1819 tablet_bootstrap.cc:492] T bb6c755fce6640d2a0b13a449d3adfb9 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:07.519574 1819 ts_tablet_manager.cc:1397] T bb6c755fce6640d2a0b13a449d3adfb9 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20250625 20:01:07.520066 1819 raft_consensus.cc:357] T bb6c755fce6640d2a0b13a449d3adfb9 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.520164 1819 raft_consensus.cc:383] T bb6c755fce6640d2a0b13a449d3adfb9 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.520193 1819 raft_consensus.cc:738] T bb6c755fce6640d2a0b13a449d3adfb9 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.520305 1819 consensus_queue.cc:260] T bb6c755fce6640d2a0b13a449d3adfb9 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.520627 1819 ts_tablet_manager.cc:1428] T bb6c755fce6640d2a0b13a449d3adfb9 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 20:01:07.520890 1827 raft_consensus.cc:491] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:07.520973 1827 raft_consensus.cc:513] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.521358 1827 leader_election.cc:290] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:07.522877 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f41fcc4d1a14c3c94a3f20115926eec" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:07.522996 1424 raft_consensus.cc:2466] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:07.523295 1497 leader_election.cc:304] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:07.523464 1935 raft_consensus.cc:2802] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:07.523504 1935 raft_consensus.cc:491] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:07.523542 1935 raft_consensus.cc:3058] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.523705 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f41fcc4d1a14c3c94a3f20115926eec" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:07.523850 1700 raft_consensus.cc:2466] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:07.524513 1935 raft_consensus.cc:513] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.524853 1935 leader_election.cc:290] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:07.524919 1817 tablet_bootstrap.cc:492] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:07.525010 1817 ts_tablet_manager.cc:1397] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250625 20:01:07.525496 1817 raft_consensus.cc:357] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.525625 1817 raft_consensus.cc:383] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:07.525658 1817 raft_consensus.cc:738] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:07.525781 1817 consensus_queue.cc:260] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.526440 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f41fcc4d1a14c3c94a3f20115926eec" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:07.526489 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f41fcc4d1a14c3c94a3f20115926eec" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:07.526537 1424 raft_consensus.cc:3058] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.526605 1700 raft_consensus.cc:3058] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.527407 1424 raft_consensus.cc:2466] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:07.527650 1700 raft_consensus.cc:2466] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:07.527796 1497 leader_election.cc:304] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:07.528008 1935 raft_consensus.cc:2802] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:07.528076 1935 raft_consensus.cc:695] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Becoming Leader. State: Replica: c73846c0665d4d53a274d053946107df, State: Running, Role: LEADER
I20250625 20:01:07.528185 1935 consensus_queue.cc:237] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.531968 1817 ts_tablet_manager.cc:1428] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250625 20:01:07.540642 1971 consensus_queue.cc:1035] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:07.544283 1971 consensus_queue.cc:1035] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:07.586668 1941 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5852fb53503b438c92f294e0c44c916d. Latest consensus state: current_term: 3 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:07.587154 1941 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:07.587280 1941 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:07.587349 1941 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:07.588611 1850 raft_consensus.cc:491] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:07.588719 1850 raft_consensus.cc:513] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.589164 1850 leader_election.cc:290] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:07.589552 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb6c755fce6640d2a0b13a449d3adfb9" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:07.589704 1561 raft_consensus.cc:2466] T bb6c755fce6640d2a0b13a449d3adfb9 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:07.590090 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb6c755fce6640d2a0b13a449d3adfb9" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:07.590215 1700 raft_consensus.cc:2466] T bb6c755fce6640d2a0b13a449d3adfb9 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:07.590767 1357 leader_election.cc:304] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:07.591132 1850 raft_consensus.cc:2802] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:07.591188 1850 raft_consensus.cc:491] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:07.591224 1850 raft_consensus.cc:3058] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.592278 1850 raft_consensus.cc:513] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.592633 1850 leader_election.cc:290] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:07.593129 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb6c755fce6640d2a0b13a449d3adfb9" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:07.593246 1700 raft_consensus.cc:3058] T bb6c755fce6640d2a0b13a449d3adfb9 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.594203 1700 raft_consensus.cc:2466] T bb6c755fce6640d2a0b13a449d3adfb9 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:07.594763 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb6c755fce6640d2a0b13a449d3adfb9" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:07.594888 1561 raft_consensus.cc:3058] T bb6c755fce6640d2a0b13a449d3adfb9 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.595818 1561 raft_consensus.cc:2466] T bb6c755fce6640d2a0b13a449d3adfb9 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:07.596211 1358 leader_election.cc:304] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:07.596659 1850 raft_consensus.cc:2802] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:07.596740 1850 raft_consensus.cc:695] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:07.596861 1850 consensus_queue.cc:237] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.606367 1980 mvcc.cc:204] Tried to move back new op lower bound from 7171611310109822976 to 7171611305716219904. Current Snapshot: MvccSnapshot[applied={T|T < 7171611310048284672}]
I20250625 20:01:07.611430 1850 consensus_queue.cc:1035] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:07.626848 1850 consensus_queue.cc:1035] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:07.707406 1820 consensus_queue.cc:1035] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:07.713706 1850 consensus_queue.cc:1035] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:07.736937 1935 consensus_queue.cc:1035] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:07.742807 1827 consensus_queue.cc:1035] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:07.793361 1083 catalog_manager.cc:3460] renamed table default.table_050aa6ac27914b98b21a7d3b966c893d [id=82357871d20c400dbb0c5727da020996] in HMS: new name default.table_8302314b76364eadb44f02cc6d58a589
I20250625 20:01:07.800371 1801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:07.800530 1801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: LEADER
W20250625 20:01:07.803125 1091 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fd28d4c5b48245c382a334b98a3066df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 20:01:07.803949 1087 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a31a093a024346aab31bc8d55a00253d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 20:01:07.804642 1089 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3629c083428e4ae0a977ffb16af83bc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 20:01:07.810046 1997 tablet.cc:1722] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 20:01:07.811932 1996 tablet.cc:1722] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 20:01:07.816305 1998 tablet.cc:1722] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 20:01:07.872570 1084 catalog_manager.cc:3460] renamed table default.table_2e6d0c334cc144fcb1166ef1dc4932f4 [id=c5f1b7b4e1e8405794eea8fc65b77fb0] in HMS: new name default.table_1d0eb88cbdfd4b16bb66bfb3d502672b
I20250625 20:01:07.933682 1085 catalog_manager.cc:3460] renamed table default.table_9b3ba4829d5b4a07b5b7610472b79b74 [id=4e3fbfc6b591464886cb1e94527676c1] in HMS: new name default.table_d0f8eb84406f4d758e34b162746daa38
I20250625 20:01:07.933825 1827 raft_consensus.cc:491] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:07.933950 1827 raft_consensus.cc:513] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.934422 1827 leader_election.cc:290] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:07.935039 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c5e0c439cdb4e67a155bd125982f971" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:07.935209 1700 raft_consensus.cc:2466] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:07.935652 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c5e0c439cdb4e67a155bd125982f971" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:07.935825 1495 leader_election.cc:304] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:07.935820 1424 raft_consensus.cc:2466] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:07.936081 1827 raft_consensus.cc:2802] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:07.936131 1827 raft_consensus.cc:491] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:07.936172 1827 raft_consensus.cc:3058] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.937207 1827 raft_consensus.cc:513] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.937566 1827 leader_election.cc:290] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:07.938055 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c5e0c439cdb4e67a155bd125982f971" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:07.938167 1424 raft_consensus.cc:3058] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.939142 1424 raft_consensus.cc:2466] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:07.939934 1497 leader_election.cc:304] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:07.940280 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c5e0c439cdb4e67a155bd125982f971" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:07.940418 1700 raft_consensus.cc:3058] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:07.941414 1700 raft_consensus.cc:2466] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:07.941857 1827 raft_consensus.cc:2802] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:07.941954 1827 raft_consensus.cc:695] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Becoming Leader. State: Replica: c73846c0665d4d53a274d053946107df, State: Running, Role: LEADER
I20250625 20:01:07.942118 1827 consensus_queue.cc:237] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:07.956707 1827 consensus_queue.cc:1035] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:07.958215 1991 consensus_queue.cc:1035] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:07.960458 1843 consensus_queue.cc:1035] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:07.966845 1935 consensus_queue.cc:1035] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:07.972004 1850 consensus_queue.cc:1035] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:07.977448 1829 consensus_queue.cc:1035] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:07.993172 1850 consensus_queue.cc:1035] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:07.997606 1086 catalog_manager.cc:3460] renamed table default.table_ae9e55cc97654931933bf1eede18454b [id=e6ca598bb22a45abb67ce2e547f3a809] in HMS: new name default.table_030bc68e40784cf2a6bd261221a0f738
I20250625 20:01:08.003072 1991 consensus_queue.cc:1035] T bb6c755fce6640d2a0b13a449d3adfb9 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:08.022096 1829 consensus_queue.cc:1035] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:08.033017 1843 consensus_queue.cc:1035] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:08.036906 1091 catalog_manager.cc:2149] created HMS catalog entry for table [id=77d32088c98540ec84ca62e7c8750cb8]
I20250625 20:01:08.045670 1680 tablet_service.cc:1468] Processing CreateTablet for tablet 21677e0e47df421c92f98ed580bbe7a7 (DEFAULT_TABLE table=default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.045985 1680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21677e0e47df421c92f98ed580bbe7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:08.046285 1541 tablet_service.cc:1468] Processing CreateTablet for tablet 21677e0e47df421c92f98ed580bbe7a7 (DEFAULT_TABLE table=default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.046557 1541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21677e0e47df421c92f98ed580bbe7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:08.047127 1091 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_d4ed5ace6c144df6af9ff3155b29c5bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:08.048046 1403 tablet_service.cc:1468] Processing CreateTablet for tablet 21677e0e47df421c92f98ed580bbe7a7 (DEFAULT_TABLE table=default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.048290 1403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21677e0e47df421c92f98ed580bbe7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:08.048424 2010 tablet_bootstrap.cc:492] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:08.049283 2010 tablet_bootstrap.cc:654] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.050154 2010 tablet_bootstrap.cc:492] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:08.050181 2013 tablet_bootstrap.cc:492] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:08.050230 2010 ts_tablet_manager.cc:1397] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:08.050706 2010 raft_consensus.cc:357] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.050814 2010 raft_consensus.cc:383] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.050850 2010 raft_consensus.cc:738] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.050966 2010 consensus_queue.cc:260] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.051069 2013 tablet_bootstrap.cc:654] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.051244 2010 ts_tablet_manager.cc:1428] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:08.051988 2013 tablet_bootstrap.cc:492] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:08.052063 2013 ts_tablet_manager.cc:1397] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:08.052488 2011 tablet_bootstrap.cc:492] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:08.052470 2013 raft_consensus.cc:357] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.052559 2013 raft_consensus.cc:383] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.052585 2013 raft_consensus.cc:738] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.052687 2013 consensus_queue.cc:260] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.052937 2013 ts_tablet_manager.cc:1428] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:08.053455 2011 tablet_bootstrap.cc:654] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.054747 2011 tablet_bootstrap.cc:492] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:08.054848 2011 ts_tablet_manager.cc:1397] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 20:01:08.055282 2011 raft_consensus.cc:357] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.055382 2011 raft_consensus.cc:383] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.055413 2011 raft_consensus.cc:738] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.055533 2011 consensus_queue.cc:260] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.055840 2011 ts_tablet_manager.cc:1428] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:08.211347 1850 raft_consensus.cc:491] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:08.211495 1850 raft_consensus.cc:513] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.212013 1850 leader_election.cc:290] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:08.212864 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21677e0e47df421c92f98ed580bbe7a7" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:08.213022 1560 raft_consensus.cc:2466] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:08.213327 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21677e0e47df421c92f98ed580bbe7a7" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:08.213477 1701 raft_consensus.cc:2466] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:08.213928 1358 leader_election.cc:304] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:08.214301 1991 raft_consensus.cc:2802] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:08.214373 1991 raft_consensus.cc:491] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:08.214421 1991 raft_consensus.cc:3058] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:08.215590 1991 raft_consensus.cc:513] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.215999 1991 leader_election.cc:290] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:08.219758 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21677e0e47df421c92f98ed580bbe7a7" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:08.219897 1700 raft_consensus.cc:3058] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:08.220942 1700 raft_consensus.cc:2466] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:08.221812 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21677e0e47df421c92f98ed580bbe7a7" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:08.221948 1560 raft_consensus.cc:3058] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:08.222971 1560 raft_consensus.cc:2466] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:08.223518 1358 leader_election.cc:304] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:08.223829 1820 raft_consensus.cc:2802] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:08.223918 1820 raft_consensus.cc:695] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:08.224063 1820 consensus_queue.cc:237] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.469655 1935 consensus_queue.cc:1035] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:08.475694 1827 consensus_queue.cc:1035] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:08.532768 1082 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_53d5923f5fec4ca8b2fb669a4cf9395a [id=224a7f2d5a4d403fa2702fd59ccbf5c9]
I20250625 20:01:08.545554 2019 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:08.545778 2019 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:08.548174 1101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "5852fb53503b438c92f294e0c44c916d" is_pre_election: true
I20250625 20:01:08.548349 2019 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:08.548565 2020 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:08.548678 2020 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:08.549396 1101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "5852fb53503b438c92f294e0c44c916d" is_pre_election: true
I20250625 20:01:08.549842 2020 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
I20250625 20:01:08.556591 1242 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" is_pre_election: true
I20250625 20:01:08.556795 1242 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f87f9e18e5b948a4a3fb3666f106abd0 in term 3.
I20250625 20:01:08.557533 1874 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0; no voters: 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:08.557844 2019 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250625 20:01:08.557933 2019 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:08.557982 2019 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 3 FOLLOWER]: Advancing to term 4
I20250625 20:01:08.560477 2019 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:08.560982 2019 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 4 election: Requested vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:08.561405 1101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "5852fb53503b438c92f294e0c44c916d"
I20250625 20:01:08.561468 1242 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "1fcdee0c0aac4013a5e3c77304b474d9"
I20250625 20:01:08.561595 1242 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 3 FOLLOWER]: Advancing to term 4
I20250625 20:01:08.562893 1242 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f87f9e18e5b948a4a3fb3666f106abd0 in term 4.
I20250625 20:01:08.562844 1895 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" is_pre_election: true
I20250625 20:01:08.563150 1895 raft_consensus.cc:2391] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fcdee0c0aac4013a5e3c77304b474d9 in current term 4: Already voted for candidate f87f9e18e5b948a4a3fb3666f106abd0 in this term.
I20250625 20:01:08.563252 1874 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0; no voters: 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:08.563478 2019 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 4 FOLLOWER]: Leader election won for term 4
I20250625 20:01:08.563573 1217 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9; no voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:08.563642 2019 raft_consensus.cc:695] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 4 LEADER]: Becoming Leader. State: Replica: f87f9e18e5b948a4a3fb3666f106abd0, State: Running, Role: LEADER
I20250625 20:01:08.563776 2020 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250625 20:01:08.564002 2019 consensus_queue.cc:237] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 3.38, Last appended by leader: 38, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:08.565258 2025 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f87f9e18e5b948a4a3fb3666f106abd0. Latest consensus state: current_term: 4 leader_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:08.565369 2025 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:08.565722 2026 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:01:08.567678 2026 catalog_manager.cc:671] Loaded metadata for table default.table_7b366e40d04d468caf65baf4c5b77004 [id=18d82881922e465cbcc90e87dfe4a5e1]
I20250625 20:01:08.568040 2026 catalog_manager.cc:671] Loaded metadata for table default.table_53d5923f5fec4ca8b2fb669a4cf9395a [id=224a7f2d5a4d403fa2702fd59ccbf5c9]
I20250625 20:01:08.568252 2026 catalog_manager.cc:671] Loaded metadata for table default.table_9b3ba4829d5b4a07b5b7610472b79b74 [id=4e3fbfc6b591464886cb1e94527676c1]
I20250625 20:01:08.568444 2026 catalog_manager.cc:671] Loaded metadata for table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]
I20250625 20:01:08.568632 2026 catalog_manager.cc:671] Loaded metadata for table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]
I20250625 20:01:08.568848 2026 catalog_manager.cc:671] Loaded metadata for table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996]
I20250625 20:01:08.569083 2026 catalog_manager.cc:671] Loaded metadata for table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583]
I20250625 20:01:08.569279 2026 catalog_manager.cc:671] Loaded metadata for table default.table_4c6973f983694e48873d7009b44c2d4d [id=af2791dc33fb464db4a937aebe316482]
I20250625 20:01:08.569466 2026 catalog_manager.cc:671] Loaded metadata for table default.table_2e6d0c334cc144fcb1166ef1dc4932f4 [id=c5f1b7b4e1e8405794eea8fc65b77fb0]
I20250625 20:01:08.569684 2026 catalog_manager.cc:671] Loaded metadata for table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]
I20250625 20:01:08.569885 2026 catalog_manager.cc:671] Loaded metadata for table default.table_ae9e55cc97654931933bf1eede18454b [id=e6ca598bb22a45abb67ce2e547f3a809]
I20250625 20:01:08.571017 2026 tablet_loader.cc:96] loaded metadata for tablet 1c5e0c439cdb4e67a155bd125982f971 (table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36])
I20250625 20:01:08.571247 2026 tablet_loader.cc:96] loaded metadata for tablet 21677e0e47df421c92f98ed580bbe7a7 (table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8])
I20250625 20:01:08.571445 2026 tablet_loader.cc:96] loaded metadata for tablet 3b7836328f054e2c95b12a8f0fe61b2b (table default.table_ae9e55cc97654931933bf1eede18454b [id=e6ca598bb22a45abb67ce2e547f3a809])
I20250625 20:01:08.571620 2026 tablet_loader.cc:96] loaded metadata for tablet 44500be7b904460fa32515082b47a6f0 (table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35])
I20250625 20:01:08.571813 2026 tablet_loader.cc:96] loaded metadata for tablet 53174bf6005e499c813b813b4a782c19 (table default.table_9b3ba4829d5b4a07b5b7610472b79b74 [id=4e3fbfc6b591464886cb1e94527676c1])
I20250625 20:01:08.571985 2026 tablet_loader.cc:96] loaded metadata for tablet 6d488e29942c46cb8c67058068ce2b52 (table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996])
I20250625 20:01:08.572458 2026 tablet_loader.cc:96] loaded metadata for tablet 7c78cfa2f9504af6910dabc002fbe1da (table default.table_53d5923f5fec4ca8b2fb669a4cf9395a [id=224a7f2d5a4d403fa2702fd59ccbf5c9])
I20250625 20:01:08.572646 2026 tablet_loader.cc:96] loaded metadata for tablet 936500cf83f84a1197231f4de02cd55b (table default.table_4c6973f983694e48873d7009b44c2d4d [id=af2791dc33fb464db4a937aebe316482])
I20250625 20:01:08.572860 2026 tablet_loader.cc:96] loaded metadata for tablet 9f41fcc4d1a14c3c94a3f20115926eec (table default.table_7b366e40d04d468caf65baf4c5b77004 [id=18d82881922e465cbcc90e87dfe4a5e1])
I20250625 20:01:08.573026 2026 tablet_loader.cc:96] loaded metadata for tablet b36de85648bb4236a9320412cdb1cfdf (table default.table_2e6d0c334cc144fcb1166ef1dc4932f4 [id=c5f1b7b4e1e8405794eea8fc65b77fb0])
I20250625 20:01:08.573280 2026 tablet_loader.cc:96] loaded metadata for tablet bb6c755fce6640d2a0b13a449d3adfb9 (table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583])
I20250625 20:01:08.573537 2026 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:01:08.573546 1087 catalog_manager.cc:2149] created HMS catalog entry for table [id=81aee385d1404c82aaeebee220cb9e68]
I20250625 20:01:08.575505 2026 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:08.575559 2026 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:01:08.576462 2026 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:01:08.576982 2026 catalog_manager.cc:5966] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Loaded TSK: 0
I20250625 20:01:08.577864 2026 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:01:08.578438 2026 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 20:01:08.578943 2026 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 11
I20250625 20:01:08.580542 1895 raft_consensus.cc:1238] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 4 LEADER]: Rejecting Update request from peer 5852fb53503b438c92f294e0c44c916d for earlier term 3. Current term is 4. Ops: [3.39-3.39]
I20250625 20:01:08.581091 2029 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 }, Status: INVALID_TERM, Last received: 3.38, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20250625 20:01:08.581233 2029 consensus_queue.cc:1230] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [LEADER]: Peer f87f9e18e5b948a4a3fb3666f106abd0 log is divergent from this leader: its last log entry 4.39 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 38
I20250625 20:01:08.581290 2029 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 3 LEADER]: Stepping down as leader of term 3
I20250625 20:01:08.581321 2029 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5852fb53503b438c92f294e0c44c916d, State: Running, Role: LEADER
I20250625 20:01:08.581528 2029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 3.39, Last appended by leader: 39, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:08.581772 2029 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 3 FOLLOWER]: Advancing to term 4
I20250625 20:01:08.584021 1242 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 4 FOLLOWER]: Rejecting Update request from peer 5852fb53503b438c92f294e0c44c916d for earlier term 3. Current term is 4. Ops: [3.39-3.39]
I20250625 20:01:08.621482 1088 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_4c6973f983694e48873d7009b44c2d4d [id=af2791dc33fb464db4a937aebe316482]
I20250625 20:01:08.640591 1089 catalog_manager.cc:2149] created HMS catalog entry for table [id=4db36ddc9a604e13af624c5e3c9565be]
W20250625 20:01:08.641386 1089 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 5852fb53503b438c92f294e0c44c916d is not leader of this config: current role FOLLOWER
I20250625 20:01:08.643440 1083 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41858 (request call id 37) took 1686 ms. Trace:
I20250625 20:01:08.643498 1085 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41858 (request call id 44) took 1565 ms. Trace:
I20250625 20:01:08.643594 1085 rpcz_store.cc:276] 0625 20:01:07.078495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:07.154945 (+ 76450us) service_pool.cc:225] Handling call
0625 20:01:07.802722 (+647777us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 20:01:08.643075 (+840353us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 20:01:08.643250 (+ 175us) catalog_manager.cc:3683] Apply alter schema
0625 20:01:08.643475 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":69770,"Hive Metastore.run_cpu_time_us":615,"Hive Metastore.run_wall_time_us":61059}
I20250625 20:01:08.643532 1083 rpcz_store.cc:276] 0625 20:01:06.957414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:06.957660 (+ 246us) service_pool.cc:225] Handling call
0625 20:01:07.498000 (+540340us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 20:01:08.642925 (+1144925us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 20:01:08.643173 (+ 248us) catalog_manager.cc:3683] Apply alter schema
0625 20:01:08.643416 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":315,"Hive Metastore.run_cpu_time_us":903,"Hive Metastore.run_wall_time_us":294785}
I20250625 20:01:08.644541 1083 master_service.cc:948] ReplaceTablet: received request to replace tablet 936500cf83f84a1197231f4de02cd55b from {username='slave'} at 127.0.0.1:41858
I20250625 20:01:08.644716 1083 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41858 (request call id 53) took 1314 ms. Trace:
I20250625 20:01:08.644759 1083 rpcz_store.cc:276] 0625 20:01:07.329776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:08.644514 (+1314738us) service_pool.cc:225] Handling call
0625 20:01:08.644705 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 20:01:08.644907 1083 master_service.cc:948] ReplaceTablet: received request to replace tablet b36de85648bb4236a9320412cdb1cfdf from {username='slave'} at 127.0.0.1:41858
I20250625 20:01:08.644981 1085 master_service.cc:948] ReplaceTablet: received request to replace tablet b36de85648bb4236a9320412cdb1cfdf from {username='slave'} at 127.0.0.1:41858
I20250625 20:01:08.645066 1083 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41858 (request call id 54) took 1315 ms. Trace:
I20250625 20:01:08.645109 1083 rpcz_store.cc:276] 0625 20:01:07.329849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:08.644891 (+1315042us) service_pool.cc:225] Handling call
0625 20:01:08.645055 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":384}
I20250625 20:01:08.645253 1083 master_service.cc:948] ReplaceTablet: received request to replace tablet 3b7836328f054e2c95b12a8f0fe61b2b from {username='slave'} at 127.0.0.1:41858
I20250625 20:01:08.645401 1083 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41858 (request call id 56) took 1285 ms. Trace:
I20250625 20:01:08.645443 1083 rpcz_store.cc:276] 0625 20:01:07.359604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:08.645237 (+1285633us) service_pool.cc:225] Handling call
0625 20:01:08.645390 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 20:01:08.645543 1083 master_service.cc:948] ReplaceTablet: received request to replace tablet 53174bf6005e499c813b813b4a782c19 from {username='slave'} at 127.0.0.1:41858
I20250625 20:01:08.645609 1086 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41858 (request call id 45) took 1534 ms. Trace:
I20250625 20:01:08.645678 1083 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41858 (request call id 57) took 1286 ms. Trace:
I20250625 20:01:08.645730 1083 rpcz_store.cc:276] 0625 20:01:07.359686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:08.645528 (+1285842us) service_pool.cc:225] Handling call
0625 20:01:08.645669 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250625 20:01:08.645954 1091 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d4ed5ace6c144df6af9ff3155b29c5bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 20:01:08.645987 1083 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41858 (request call id 58) took 1273 ms. Trace:
I20250625 20:01:08.646034 1083 rpcz_store.cc:276] 0625 20:01:07.372499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:08.645838 (+1273339us) service_pool.cc:225] Handling call
0625 20:01:08.645974 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 20:01:08.646307 1083 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41858 (request call id 59) took 1260 ms. Trace:
I20250625 20:01:08.646353 1083 rpcz_store.cc:276] 0625 20:01:07.385950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:08.646143 (+1260193us) service_pool.cc:225] Handling call
0625 20:01:08.646296 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 20:01:08.646590 1082 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41858 (request call id 47) took 1464 ms. Trace:
I20250625 20:01:08.646651 1082 rpcz_store.cc:276] 0625 20:01:07.181751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:07.260278 (+ 78527us) service_pool.cc:225] Handling call
0625 20:01:07.803637 (+543359us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 20:01:08.646578 (+842941us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":232992,"Hive Metastore.run_cpu_time_us":235,"Hive Metastore.run_wall_time_us":495915}
I20250625 20:01:08.646955 1082 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41858 (request call id 61) took 1185 ms. Trace:
I20250625 20:01:08.647004 1082 rpcz_store.cc:276] 0625 20:01:07.461418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:08.646804 (+1185386us) service_pool.cc:225] Handling call
0625 20:01:08.646945 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 20:01:08.647241 1082 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41858 (request call id 62) took 1154 ms. Trace:
I20250625 20:01:08.647286 1082 rpcz_store.cc:276] 0625 20:01:07.492442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:08.647102 (+1154660us) service_pool.cc:225] Handling call
0625 20:01:08.647231 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 20:01:08.647512 1082 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41858 (request call id 63) took 1136 ms. Trace:
I20250625 20:01:08.647559 1082 rpcz_store.cc:276] 0625 20:01:07.510956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:08.647369 (+1136413us) service_pool.cc:225] Handling call
0625 20:01:08.647502 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 20:01:08.647871 1082 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41858 (request call id 64) took 1075 ms. Trace:
I20250625 20:01:08.647918 1082 rpcz_store.cc:276] 0625 20:01:07.572876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:08.647673 (+1074797us) service_pool.cc:225] Handling call
0625 20:01:08.647858 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250625 20:01:08.647943 1755 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.204.62:36889)
I20250625 20:01:08.648517 1082 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:41862 (request call id 0) took 1278 ms. Trace:
I20250625 20:01:08.648555 1082 rpcz_store.cc:276] 0625 20:01:07.369691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:08.648417 (+1278726us) service_pool.cc:225] Handling call
0625 20:01:08.648509 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250625 20:01:08.648571 1755 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.204.62:36889)
I20250625 20:01:08.648682 1088 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41858 (request call id 49) took 1387 ms. Trace:
W20250625 20:01:08.648746 1755 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.204.62:36889)
I20250625 20:01:08.648733 1088 rpcz_store.cc:276] 0625 20:01:07.260935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:07.416109 (+155174us) service_pool.cc:225] Handling call
0625 20:01:07.804403 (+388294us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 20:01:08.648675 (+844272us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":769046,"Hive Metastore.run_cpu_time_us":202,"Hive Metastore.run_wall_time_us":47914}
I20250625 20:01:08.645663 1086 rpcz_store.cc:276] 0625 20:01:07.110921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:07.157346 (+ 46425us) service_pool.cc:225] Handling call
0625 20:01:07.802823 (+645477us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 20:01:08.645217 (+842394us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 20:01:08.645371 (+ 154us) catalog_manager.cc:3683] Apply alter schema
0625 20:01:08.645597 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":130775,"Hive Metastore.run_cpu_time_us":557,"Hive Metastore.run_wall_time_us":63861}
I20250625 20:01:08.646595 1083 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41858 (request call id 60) took 1197 ms. Trace:
I20250625 20:01:08.650460 1083 rpcz_store.cc:276] 0625 20:01:07.448698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:08.646472 (+1197774us) service_pool.cc:225] Handling call
0625 20:01:08.646583 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 20:01:08.643440 1084 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41858 (request call id 38) took 1654 ms. Trace:
I20250625 20:01:08.653302 1084 rpcz_store.cc:276] 0625 20:01:06.988785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:06.988825 (+ 40us) service_pool.cc:225] Handling call
0625 20:01:07.802573 (+813748us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 20:01:08.643016 (+840443us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 20:01:08.643173 (+ 157us) catalog_manager.cc:3683] Apply alter schema
0625 20:01:08.643416 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":5772,"Hive Metastore.run_cpu_time_us":527,"Hive Metastore.run_wall_time_us":64056}
W20250625 20:01:08.655442 1755 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.204.62:36889)
I20250625 20:01:08.659049 1881 master_service.cc:948] ReplaceTablet: received request to replace tablet b36de85648bb4236a9320412cdb1cfdf from {username='slave'} at 127.0.0.1:49704
I20250625 20:01:08.660373 1886 master_service.cc:948] ReplaceTablet: received request to replace tablet 3b7836328f054e2c95b12a8f0fe61b2b from {username='slave'} at 127.0.0.1:49704
I20250625 20:01:08.660567 1885 master_service.cc:948] ReplaceTablet: received request to replace tablet 53174bf6005e499c813b813b4a782c19 from {username='slave'} at 127.0.0.1:49704
I20250625 20:01:08.661010 1888 master_service.cc:948] ReplaceTablet: received request to replace tablet 936500cf83f84a1197231f4de02cd55b from {username='slave'} at 127.0.0.1:49704
I20250625 20:01:08.664824 1879 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49704:
name: "default.Table_063972c2ee4b4e468cf982177f09f91a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:08.666249 1889 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49704:
name: "default.Table_01c13560571f419e9b6f45edee3a2a37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:08.667407 1883 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49704:
table { table_name: "default.Table_60554f72b1aa48aa93aaea511a985fe7" } modify_external_catalogs: true
I20250625 20:01:08.667570 1883 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49704:
table { table_name: "default.Table_60554f72b1aa48aa93aaea511a985fe7" } modify_external_catalogs: true
I20250625 20:01:08.667940 1882 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49704:
name: "default.Table_03571335de5b4e4fb1fb01b5e6f4a6fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:08.668498 1884 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49704:
name: "default.Table_c1154a92402c4c0485b49679f754ecb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:08.668823 1887 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49704:
table { table_name: "default.Table_976af21482a74bc79eb800684cce978e" } new_table_name: "default.Table_ad0e628a3e004a9da4a42cb7d6051e3e" modify_external_catalogs: true
I20250625 20:01:08.669184 1991 consensus_queue.cc:1035] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:08.645133 1085 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41858 (request call id 55) took 1286 ms. Trace:
I20250625 20:01:08.669397 1085 rpcz_store.cc:276] 0625 20:01:07.358305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:08.644961 (+1286656us) service_pool.cc:225] Handling call
0625 20:01:08.645126 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 20:01:08.674428 1242 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 4 FOLLOWER]: Refusing update from remote peer f87f9e18e5b948a4a3fb3666f106abd0: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 4 index: 40. (index mismatch)
I20250625 20:01:08.674551 1101 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 4 FOLLOWER]: Refusing update from remote peer f87f9e18e5b948a4a3fb3666f106abd0: Log matching property violated. Preceding OpId in replica: term: 3 index: 39. Preceding OpId from leader: term: 4 index: 40. (index mismatch)
I20250625 20:01:08.675040 2019 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20250625 20:01:08.675395 2019 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20250625 20:01:08.675478 2019 consensus_queue.cc:1230] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [LEADER]: Peer 5852fb53503b438c92f294e0c44c916d log is divergent from this leader: its last log entry 3.39 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 38
I20250625 20:01:08.676826 1101 pending_rounds.cc:85] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Aborting all ops after (but not including) 38
I20250625 20:01:08.676913 1101 pending_rounds.cc:107] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Aborting uncommitted WRITE_OP operation due to leader change: 3.39
I20250625 20:01:08.677188 2020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f87f9e18e5b948a4a3fb3666f106abd0. Latest consensus state: current_term: 4 leader_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:08.677284 2020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: FOLLOWER
W20250625 20:01:08.677708 1087 catalog_manager.cc:2177] Aborted: an error occurred while writing to the sys-catalog: Op aborted by new leader
I20250625 20:01:08.678543 2030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f87f9e18e5b948a4a3fb3666f106abd0. Latest consensus state: current_term: 4 leader_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:08.678613 2030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:08.679797 2019 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:08.679895 2019 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:08.680052 2019 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:08.680123 2019 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:08.680078 2020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:08.680157 2020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:08.681806 2030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:08.681905 2030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:08.684608 1820 consensus_queue.cc:1035] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:08.691490 1885 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Sending DeleteTablet for 3 replicas of tablet 53174bf6005e499c813b813b4a782c19
I20250625 20:01:08.691613 1888 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Sending DeleteTablet for 3 replicas of tablet 936500cf83f84a1197231f4de02cd55b
I20250625 20:01:08.692297 1881 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Sending DeleteTablet for 3 replicas of tablet b36de85648bb4236a9320412cdb1cfdf
I20250625 20:01:08.692777 1881 catalog_manager.cc:6652] ReplaceTablet: tablet b36de85648bb4236a9320412cdb1cfdf deleted and replaced by tablet 07686deb2e184c9fbae06fef5365d0d5
I20250625 20:01:08.693428 1886 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Sending DeleteTablet for 3 replicas of tablet 3b7836328f054e2c95b12a8f0fe61b2b
I20250625 20:01:08.693631 1886 catalog_manager.cc:6652] ReplaceTablet: tablet 3b7836328f054e2c95b12a8f0fe61b2b deleted and replaced by tablet b78b6f29890140de958cebed9868b303
I20250625 20:01:08.693953 1881 catalog_manager.cc:5582] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df reported cstate change: term changed from 0 to 1, leader changed from <none> to 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1). New cstate: current_term: 1 leader_uuid: "27aa5b53b16747f9a72485aaf2417b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } }
I20250625 20:01:08.699059 1888 catalog_manager.cc:6652] ReplaceTablet: tablet 936500cf83f84a1197231f4de02cd55b deleted and replaced by tablet ac2af97f75484afab32084c274d890b3
I20250625 20:01:08.700755 1885 catalog_manager.cc:6652] ReplaceTablet: tablet 53174bf6005e499c813b813b4a782c19 deleted and replaced by tablet d5d6a94383be4a1dba614c0db1b88003
I20250625 20:01:08.700991 1885 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49704:
name: "default.Table_312eaaab0028455bb53f1af2373b85ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:08.701550 1881 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49704:
name: "default.Table_967fd289bea3427c81a71d7bce689c47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:08.702279 1886 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49704:
name: "default.Table_efbb957abb8f444ea073162901fa6215"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:08.704953 1609 heartbeater.cc:499] Master 127.0.204.61:38005 was elected leader, sending a full tablet report...
I20250625 20:01:08.711769 1089 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=4db36ddc9a604e13af624c5e3c9565be]
I20250625 20:01:08.712081 1089 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41858 (request call id 50) took 1436 ms. Trace:
I20250625 20:01:08.712142 1089 rpcz_store.cc:276] 0625 20:01:07.275874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:07.447551 (+171677us) service_pool.cc:225] Handling call
0625 20:01:07.804716 (+357165us) catalog_manager.cc:2076] Acquired catalog manager lock
0625 20:01:07.805005 (+ 289us) catalog_manager.cc:2122] Created new table and tablet info
0625 20:01:08.640579 (+835574us) catalog_manager.cc:2148] Created new table in HMS catalog
0625 20:01:08.641279 (+ 700us) write_op.cc:489] Releasing partition, row and schema locks
0625 20:01:08.641281 (+ 2us) write_op.cc:454] Released schema lock
0625 20:01:08.641480 (+ 199us) catalog_manager.cc:2156] Rolling back HMS table creation
0625 20:01:08.712067 (+ 70587us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 20:01:08.641190 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 5852fb53503b438c92f294e0c44c916d is not leader of this config: current role FOLLOWER)
0625 20:01:08.641202 (+ 12us) write_op.cc:489] Releasing partition, row and schema locks
0625 20:01:08.641207 (+ 5us) write_op.cc:454] Released schema lock
0625 20:01:08.641210 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":817643,"Hive Metastore.run_cpu_time_us":435,"Hive Metastore.run_wall_time_us":87951,"child_traces":[["op",{}]]}
I20250625 20:01:08.712983 1888 catalog_manager.cc:5582] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3). New cstate: current_term: 1 leader_uuid: "0ed894f2ff26414190d33cd134c36bc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } }
I20250625 20:01:08.713666 1888 catalog_manager.cc:5582] T bb6c755fce6640d2a0b13a449d3adfb9 P c73846c0665d4d53a274d053946107df reported cstate change: term changed from 0 to 1, leader changed from <none> to 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1). New cstate: current_term: 1 leader_uuid: "27aa5b53b16747f9a72485aaf2417b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } }
I20250625 20:01:08.714663 1888 catalog_manager.cc:5582] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df reported cstate change: term changed from 0 to 1, leader changed from <none> to c73846c0665d4d53a274d053946107df (127.0.204.2). New cstate: current_term: 1 leader_uuid: "c73846c0665d4d53a274d053946107df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } health_report { overall_health: UNKNOWN } } }
I20250625 20:01:08.715081 1888 catalog_manager.cc:5582] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df reported cstate change: term changed from 0 to 1, leader changed from <none> to c73846c0665d4d53a274d053946107df (127.0.204.2). New cstate: current_term: 1 leader_uuid: "c73846c0665d4d53a274d053946107df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } health_report { overall_health: HEALTHY } } }
W20250625 20:01:08.718664 1954 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07686deb2e184c9fbae06fef5365d0d5 consensus state has no leader
W20250625 20:01:08.732689 1954 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5d6a94383be4a1dba614c0db1b88003 consensus state has no leader
I20250625 20:01:08.734354 1888 catalog_manager.cc:6127] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996] alter complete (version 1)
W20250625 20:01:08.738690 1954 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b78b6f29890140de958cebed9868b303 consensus state has no leader
I20250625 20:01:08.739174 1954 catalog_manager.cc:2747] Deleting table default.table_53d5923f5fec4ca8b2fb669a4cf9395a [id=224a7f2d5a4d403fa2702fd59ccbf5c9] in response to Hive Metastore notification log event 16
I20250625 20:01:08.740361 1403 tablet_service.cc:1515] Processing DeleteTablet for tablet 53174bf6005e499c813b813b4a782c19 with delete_type TABLET_DATA_DELETED (replaced by tablet d5d6a94383be4a1dba614c0db1b88003) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:08.740669 1404 tablet_service.cc:1515] Processing DeleteTablet for tablet b36de85648bb4236a9320412cdb1cfdf with delete_type TABLET_DATA_DELETED (replaced by tablet 07686deb2e184c9fbae06fef5365d0d5) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:08.740828 1402 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b7836328f054e2c95b12a8f0fe61b2b with delete_type TABLET_DATA_DELETED (replaced by tablet b78b6f29890140de958cebed9868b303) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:08.740934 1401 tablet_service.cc:1515] Processing DeleteTablet for tablet 936500cf83f84a1197231f4de02cd55b with delete_type TABLET_DATA_DELETED (replaced by tablet ac2af97f75484afab32084c274d890b3) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:08.741240 1400 tablet_service.cc:1468] Processing CreateTablet for tablet b78b6f29890140de958cebed9868b303 (DEFAULT_TABLE table=default.table_ae9e55cc97654931933bf1eede18454b [id=e6ca598bb22a45abb67ce2e547f3a809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.741526 1400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b78b6f29890140de958cebed9868b303. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:08.742357 1888 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49704:
table { table_name: "default.Table_ef511cc05ad8422e98275bbb9b82ecb4" } modify_external_catalogs: true
I20250625 20:01:08.742491 1888 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49704:
table { table_name: "default.Table_ef511cc05ad8422e98275bbb9b82ecb4" } modify_external_catalogs: true
I20250625 20:01:08.743929 1399 tablet_service.cc:1468] Processing CreateTablet for tablet 07686deb2e184c9fbae06fef5365d0d5 (DEFAULT_TABLE table=default.table_2e6d0c334cc144fcb1166ef1dc4932f4 [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.744248 1398 tablet_service.cc:1468] Processing CreateTablet for tablet ac2af97f75484afab32084c274d890b3 (DEFAULT_TABLE table=default.table_4c6973f983694e48873d7009b44c2d4d [id=af2791dc33fb464db4a937aebe316482]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.744514 1397 tablet_service.cc:1468] Processing CreateTablet for tablet d5d6a94383be4a1dba614c0db1b88003 (DEFAULT_TABLE table=default.table_9b3ba4829d5b4a07b5b7610472b79b74 [id=4e3fbfc6b591464886cb1e94527676c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.745178 1397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5d6a94383be4a1dba614c0db1b88003. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:08.747538 1398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac2af97f75484afab32084c274d890b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:08.749608 1399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07686deb2e184c9fbae06fef5365d0d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 20:01:08.751219 1872 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 07686deb2e184c9fbae06fef5365d0d5 (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 07686deb2e184c9fbae06fef5365d0d5 consensus state has no leader
I20250625 20:01:08.753974 1954 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Sending DeleteTablet for 3 replicas of tablet 7c78cfa2f9504af6910dabc002fbe1da
I20250625 20:01:08.754969 1399 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c78cfa2f9504af6910dabc002fbe1da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:08.755019 1954 catalog_manager.cc:2747] Deleting table default.table_4c6973f983694e48873d7009b44c2d4d [id=af2791dc33fb464db4a937aebe316482] in response to Hive Metastore notification log event 18
I20250625 20:01:08.760628 1954 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Sending DeleteTablet for 3 replicas of tablet ac2af97f75484afab32084c274d890b3
I20250625 20:01:08.761191 1399 tablet_service.cc:1515] Processing DeleteTablet for tablet ac2af97f75484afab32084c274d890b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:08.763794 1540 tablet_service.cc:1515] Processing DeleteTablet for tablet b36de85648bb4236a9320412cdb1cfdf with delete_type TABLET_DATA_DELETED (replaced by tablet 07686deb2e184c9fbae06fef5365d0d5) from {username='slave'} at 127.0.0.1:57380
I20250625 20:01:08.764112 1539 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b7836328f054e2c95b12a8f0fe61b2b with delete_type TABLET_DATA_DELETED (replaced by tablet b78b6f29890140de958cebed9868b303) from {username='slave'} at 127.0.0.1:57380
I20250625 20:01:08.764243 1538 tablet_service.cc:1515] Processing DeleteTablet for tablet 936500cf83f84a1197231f4de02cd55b with delete_type TABLET_DATA_DELETED (replaced by tablet ac2af97f75484afab32084c274d890b3) from {username='slave'} at 127.0.0.1:57380
I20250625 20:01:08.764513 1537 tablet_service.cc:1468] Processing CreateTablet for tablet b78b6f29890140de958cebed9868b303 (DEFAULT_TABLE table=default.table_ae9e55cc97654931933bf1eede18454b [id=e6ca598bb22a45abb67ce2e547f3a809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.764830 1537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b78b6f29890140de958cebed9868b303. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 20:01:08.766206 1879 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_063972c2ee4b4e468cf982177f09f91a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 20:01:08.768002 1536 tablet_service.cc:1468] Processing CreateTablet for tablet 07686deb2e184c9fbae06fef5365d0d5 (DEFAULT_TABLE table=default.table_2e6d0c334cc144fcb1166ef1dc4932f4 [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.768291 1536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07686deb2e184c9fbae06fef5365d0d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:08.769236 1954 catalog_manager.cc:2747] Deleting table default.table_3629c083428e4ae0a977ffb16af83bc6 [id=4db36ddc9a604e13af624c5e3c9565be] in response to Hive Metastore notification log event 20
W20250625 20:01:08.769435 1954 hms_notification_log_listener.cc:329] Failed to handle Hive Metastore notification: 20 DROP_TABLE default.table_3629c083428e4ae0a977ffb16af83bc6: Not found: the table does not exist: table_id: "4db36ddc9a604e13af624c5e3c9565be" table_name: "default.table_3629c083428e4ae0a977ffb16af83bc6"
I20250625 20:01:08.769770 1954 catalog_manager.cc:2747] Deleting table default.table_7b366e40d04d468caf65baf4c5b77004 [id=18d82881922e465cbcc90e87dfe4a5e1] in response to Hive Metastore notification log event 21
I20250625 20:01:08.771008 2058 tablet_bootstrap.cc:492] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:08.771152 1535 tablet_service.cc:1468] Processing CreateTablet for tablet ac2af97f75484afab32084c274d890b3 (DEFAULT_TABLE table=default.table_4c6973f983694e48873d7009b44c2d4d [id=af2791dc33fb464db4a937aebe316482]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.771399 1535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac2af97f75484afab32084c274d890b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:08.771939 1534 tablet_service.cc:1468] Processing CreateTablet for tablet d5d6a94383be4a1dba614c0db1b88003 (DEFAULT_TABLE table=default.table_9b3ba4829d5b4a07b5b7610472b79b74 [id=4e3fbfc6b591464886cb1e94527676c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.772178 1534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5d6a94383be4a1dba614c0db1b88003. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:08.774255 1532 tablet_service.cc:1515] Processing DeleteTablet for tablet ac2af97f75484afab32084c274d890b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:57380
I20250625 20:01:08.774408 1533 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c78cfa2f9504af6910dabc002fbe1da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:57380
I20250625 20:01:08.774513 1090 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_7b366e40d04d468caf65baf4c5b77004 [id=18d82881922e465cbcc90e87dfe4a5e1]
I20250625 20:01:08.774786 1541 tablet_service.cc:1515] Processing DeleteTablet for tablet 53174bf6005e499c813b813b4a782c19 with delete_type TABLET_DATA_DELETED (replaced by tablet d5d6a94383be4a1dba614c0db1b88003) from {username='slave'} at 127.0.0.1:57380
I20250625 20:01:08.774933 1090 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41858 (request call id 51) took 1466 ms. Trace:
I20250625 20:01:08.774986 1090 rpcz_store.cc:276] 0625 20:01:07.308384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:07.518212 (+209828us) service_pool.cc:225] Handling call
0625 20:01:08.642851 (+1124639us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 20:01:08.774917 (+132066us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":68806,"Hive Metastore.run_cpu_time_us":127,"Hive Metastore.run_wall_time_us":62155}
I20250625 20:01:08.775123 2058 tablet_bootstrap.cc:654] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.776628 2057 tablet_replica.cc:331] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:08.776831 2057 raft_consensus.cc:2241] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:08.777045 2057 raft_consensus.cc:2270] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.777742 2057 ts_tablet_manager.cc:1905] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.779197 1954 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Sending DeleteTablet for 3 replicas of tablet 9f41fcc4d1a14c3c94a3f20115926eec
W20250625 20:01:08.779981 1889 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_01c13560571f419e9b6f45edee3a2a37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 20:01:08.779985 2057 ts_tablet_manager.cc:1918] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.780133 2057 log.cc:1199] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/b36de85648bb4236a9320412cdb1cfdf
I20250625 20:01:08.780465 2057 ts_tablet_manager.cc:1939] T b36de85648bb4236a9320412cdb1cfdf P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
W20250625 20:01:08.780905 1882 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 20:01:08.781308 2057 tablet_replica.cc:331] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:08.781481 2057 raft_consensus.cc:2241] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
W20250625 20:01:08.781548 1884 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c1154a92402c4c0485b49679f754ecb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 20:01:08.781680 2057 raft_consensus.cc:2270] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.781889 1680 tablet_service.cc:1515] Processing DeleteTablet for tablet b36de85648bb4236a9320412cdb1cfdf with delete_type TABLET_DATA_DELETED (replaced by tablet 07686deb2e184c9fbae06fef5365d0d5) from {username='slave'} at 127.0.0.1:43128
I20250625 20:01:08.782147 2057 ts_tablet_manager.cc:1905] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.782167 1681 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b7836328f054e2c95b12a8f0fe61b2b with delete_type TABLET_DATA_DELETED (replaced by tablet b78b6f29890140de958cebed9868b303) from {username='slave'} at 127.0.0.1:43128
I20250625 20:01:08.782305 1679 tablet_service.cc:1515] Processing DeleteTablet for tablet 936500cf83f84a1197231f4de02cd55b with delete_type TABLET_DATA_DELETED (replaced by tablet ac2af97f75484afab32084c274d890b3) from {username='slave'} at 127.0.0.1:43128
W20250625 20:01:08.782333 1885 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_312eaaab0028455bb53f1af2373b85ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 20:01:08.782425 1678 tablet_service.cc:1515] Processing DeleteTablet for tablet 53174bf6005e499c813b813b4a782c19 with delete_type TABLET_DATA_DELETED (replaced by tablet d5d6a94383be4a1dba614c0db1b88003) from {username='slave'} at 127.0.0.1:43128
I20250625 20:01:08.782711 1677 tablet_service.cc:1468] Processing CreateTablet for tablet b78b6f29890140de958cebed9868b303 (DEFAULT_TABLE table=default.table_ae9e55cc97654931933bf1eede18454b [id=e6ca598bb22a45abb67ce2e547f3a809]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 20:01:08.782943 1881 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_967fd289bea3427c81a71d7bce689c47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 20:01:08.782999 1677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b78b6f29890140de958cebed9868b303. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 20:01:08.783547 1886 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_efbb957abb8f444ea073162901fa6215 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 20:01:08.784292 1871 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet b36de85648bb4236a9320412cdb1cfdf (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]) successfully deleted
I20250625 20:01:08.784492 1537 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f41fcc4d1a14c3c94a3f20115926eec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:57380
I20250625 20:01:08.784816 1399 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f41fcc4d1a14c3c94a3f20115926eec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:08.785405 2057 ts_tablet_manager.cc:1918] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.785517 2057 log.cc:1199] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/3b7836328f054e2c95b12a8f0fe61b2b
I20250625 20:01:08.785573 1675 tablet_service.cc:1468] Processing CreateTablet for tablet 07686deb2e184c9fbae06fef5365d0d5 (DEFAULT_TABLE table=default.table_2e6d0c334cc144fcb1166ef1dc4932f4 [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.785842 1675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07686deb2e184c9fbae06fef5365d0d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:08.786062 2060 tablet_replica.cc:331] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:08.785862 2057 ts_tablet_manager.cc:1939] T 3b7836328f054e2c95b12a8f0fe61b2b P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:08.786303 2060 raft_consensus.cc:2241] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:08.787000 1871 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet 3b7836328f054e2c95b12a8f0fe61b2b (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]) successfully deleted
I20250625 20:01:08.787218 2060 raft_consensus.cc:2270] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.787422 2057 tablet_replica.cc:331] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:08.787535 2057 raft_consensus.cc:2241] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:08.787639 2057 raft_consensus.cc:2270] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.788105 2060 ts_tablet_manager.cc:1905] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.788122 1676 tablet_service.cc:1468] Processing CreateTablet for tablet ac2af97f75484afab32084c274d890b3 (DEFAULT_TABLE table=default.table_4c6973f983694e48873d7009b44c2d4d [id=af2791dc33fb464db4a937aebe316482]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.788388 1676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac2af97f75484afab32084c274d890b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:08.789801 2060 ts_tablet_manager.cc:1918] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.789870 2060 log.cc:1199] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/b36de85648bb4236a9320412cdb1cfdf
I20250625 20:01:08.790086 2060 ts_tablet_manager.cc:1939] T b36de85648bb4236a9320412cdb1cfdf P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:08.790395 2061 tablet_bootstrap.cc:492] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:08.790575 1674 tablet_service.cc:1468] Processing CreateTablet for tablet d5d6a94383be4a1dba614c0db1b88003 (DEFAULT_TABLE table=default.table_9b3ba4829d5b4a07b5b7610472b79b74 [id=4e3fbfc6b591464886cb1e94527676c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.790781 2060 tablet_replica.cc:331] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:08.790849 1674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5d6a94383be4a1dba614c0db1b88003. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:08.790988 2060 raft_consensus.cc:2241] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:08.791133 2060 raft_consensus.cc:2270] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.791271 1673 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c78cfa2f9504af6910dabc002fbe1da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:43128
I20250625 20:01:08.791450 1672 tablet_service.cc:1515] Processing DeleteTablet for tablet ac2af97f75484afab32084c274d890b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:43128
I20250625 20:01:08.791527 2061 tablet_bootstrap.cc:654] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.791646 2060 ts_tablet_manager.cc:1905] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.792687 2058 tablet_bootstrap.cc:492] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:08.792793 2058 ts_tablet_manager.cc:1397] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20250625 20:01:08.793365 2058 raft_consensus.cc:357] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.793488 2058 raft_consensus.cc:383] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.793525 2058 raft_consensus.cc:738] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.793673 2058 consensus_queue.cc:260] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.794057 2060 ts_tablet_manager.cc:1918] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.794162 2060 log.cc:1199] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/3b7836328f054e2c95b12a8f0fe61b2b
I20250625 20:01:08.794435 2060 ts_tablet_manager.cc:1939] T 3b7836328f054e2c95b12a8f0fe61b2b P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:08.795046 2060 tablet_replica.cc:331] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:08.795253 2060 raft_consensus.cc:2241] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:08.795604 2060 raft_consensus.cc:2270] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.796195 2060 ts_tablet_manager.cc:1905] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.796262 1872 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet b36de85648bb4236a9320412cdb1cfdf (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]) successfully deleted
I20250625 20:01:08.796396 1872 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet 3b7836328f054e2c95b12a8f0fe61b2b (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]) successfully deleted
I20250625 20:01:08.796576 1672 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f41fcc4d1a14c3c94a3f20115926eec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:43128
I20250625 20:01:08.796691 2058 ts_tablet_manager.cc:1428] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 20:01:08.798202 2060 ts_tablet_manager.cc:1918] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.798314 2060 log.cc:1199] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/936500cf83f84a1197231f4de02cd55b
I20250625 20:01:08.798600 2060 ts_tablet_manager.cc:1939] T 936500cf83f84a1197231f4de02cd55b P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:08.798769 2058 tablet_bootstrap.cc:492] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:08.799199 2060 tablet_replica.cc:331] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:08.799361 2060 raft_consensus.cc:2241] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:08.799582 2060 raft_consensus.cc:2270] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.799818 2057 ts_tablet_manager.cc:1905] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.800034 2060 ts_tablet_manager.cc:1905] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.800305 2058 tablet_bootstrap.cc:654] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.801864 2058 tablet_bootstrap.cc:492] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:08.801957 2058 ts_tablet_manager.cc:1397] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 20:01:08.802222 2060 ts_tablet_manager.cc:1918] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.802332 2060 log.cc:1199] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/53174bf6005e499c813b813b4a782c19
I20250625 20:01:08.802399 2057 ts_tablet_manager.cc:1918] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.802470 2057 log.cc:1199] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/936500cf83f84a1197231f4de02cd55b
I20250625 20:01:08.802424 2058 raft_consensus.cc:357] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.802515 2058 raft_consensus.cc:383] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.802569 2058 raft_consensus.cc:738] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.802695 2057 ts_tablet_manager.cc:1939] T 936500cf83f84a1197231f4de02cd55b P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:08.802735 2058 consensus_queue.cc:260] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.802888 2060 ts_tablet_manager.cc:1939] T 53174bf6005e499c813b813b4a782c19 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:08.803472 2060 tablet_replica.cc:331] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:08.803622 2060 raft_consensus.cc:2241] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:08.803824 2060 raft_consensus.cc:2270] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.803941 1872 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet 53174bf6005e499c813b813b4a782c19 (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]) successfully deleted
I20250625 20:01:08.804263 2060 ts_tablet_manager.cc:1905] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.804728 2058 ts_tablet_manager.cc:1428] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20250625 20:01:08.805008 1871 catalog_manager.cc:4908] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): delete failed for tablet ac2af97f75484afab32084c274d890b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac2af97f75484afab32084c274d890b3 already in progress: creating tablet
I20250625 20:01:08.806056 2060 ts_tablet_manager.cc:1918] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.806201 2060 log.cc:1199] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/7c78cfa2f9504af6910dabc002fbe1da
I20250625 20:01:08.806488 2060 ts_tablet_manager.cc:1939] T 7c78cfa2f9504af6910dabc002fbe1da P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:08.807065 2058 tablet_bootstrap.cc:492] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:08.807798 1872 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet 7c78cfa2f9504af6910dabc002fbe1da (table default.table_53d5923f5fec4ca8b2fb669a4cf9395a [id=224a7f2d5a4d403fa2702fd59ccbf5c9]) successfully deleted
I20250625 20:01:08.808027 2057 tablet_replica.cc:331] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:08.808146 2057 raft_consensus.cc:2241] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:08.808255 2057 raft_consensus.cc:2270] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.808427 2058 tablet_bootstrap.cc:654] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
W20250625 20:01:08.808597 1872 catalog_manager.cc:4908] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): delete failed for tablet ac2af97f75484afab32084c274d890b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac2af97f75484afab32084c274d890b3 already in progress: creating tablet
I20250625 20:01:08.808622 2060 tablet_replica.cc:331] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:08.808841 2060 raft_consensus.cc:2241] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:08.808964 2056 tablet_bootstrap.cc:492] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:08.809149 2060 raft_consensus.cc:2270] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.809675 2061 tablet_bootstrap.cc:492] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:08.809767 2061 ts_tablet_manager.cc:1397] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20250625 20:01:08.809180 2055 tablet_replica.cc:331] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:08.809973 2055 raft_consensus.cc:2241] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:08.810248 2061 raft_consensus.cc:357] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.810356 2061 raft_consensus.cc:383] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.810390 2061 raft_consensus.cc:738] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.810547 2061 consensus_queue.cc:260] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.810580 2055 raft_consensus.cc:2270] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.810809 1749 heartbeater.cc:499] Master 127.0.204.61:38005 was elected leader, sending a full tablet report...
I20250625 20:01:08.811587 2061 ts_tablet_manager.cc:1428] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 20:01:08.812270 2061 tablet_bootstrap.cc:492] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:08.813710 2061 tablet_bootstrap.cc:654] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.815156 2057 ts_tablet_manager.cc:1905] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.815310 2060 ts_tablet_manager.cc:1905] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.816044 2055 ts_tablet_manager.cc:1905] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.817917 2060 ts_tablet_manager.cc:1918] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.817999 2060 log.cc:1199] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/9f41fcc4d1a14c3c94a3f20115926eec
I20250625 20:01:08.818279 2060 ts_tablet_manager.cc:1939] T 9f41fcc4d1a14c3c94a3f20115926eec P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:08.819027 1872 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet 9f41fcc4d1a14c3c94a3f20115926eec (table default.table_7b366e40d04d468caf65baf4c5b77004 [id=18d82881922e465cbcc90e87dfe4a5e1]) successfully deleted
I20250625 20:01:08.816855 2056 tablet_bootstrap.cc:654] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.824616 2061 tablet_bootstrap.cc:492] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:08.824738 2061 ts_tablet_manager.cc:1397] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250625 20:01:08.825273 2061 raft_consensus.cc:357] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.825393 2061 raft_consensus.cc:383] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.825424 2061 raft_consensus.cc:738] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.825556 2061 consensus_queue.cc:260] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.825826 2061 ts_tablet_manager.cc:1428] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250625 20:01:08.826042 2061 tablet_bootstrap.cc:492] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:08.826936 2057 ts_tablet_manager.cc:1918] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.827041 2057 log.cc:1199] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/7c78cfa2f9504af6910dabc002fbe1da
I20250625 20:01:08.827075 2061 tablet_bootstrap.cc:654] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.827282 2057 ts_tablet_manager.cc:1939] T 7c78cfa2f9504af6910dabc002fbe1da P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:08.828683 2058 tablet_bootstrap.cc:492] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:08.828790 2058 ts_tablet_manager.cc:1397] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20250625 20:01:08.828927 2055 ts_tablet_manager.cc:1918] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.828997 2055 log.cc:1199] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/53174bf6005e499c813b813b4a782c19
I20250625 20:01:08.829277 2055 ts_tablet_manager.cc:1939] T 53174bf6005e499c813b813b4a782c19 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:08.829335 2058 raft_consensus.cc:357] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.829447 2058 raft_consensus.cc:383] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.829480 2058 raft_consensus.cc:738] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.829612 2058 consensus_queue.cc:260] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.829905 2055 tablet_replica.cc:331] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:08.829902 2058 ts_tablet_manager.cc:1428] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250625 20:01:08.830032 2055 raft_consensus.cc:2241] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Raft consensus shutting down.
W20250625 20:01:08.830050 1358 consensus_peers.cc:487] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 -> Peer 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): Couldn't send request to peer 0ed894f2ff26414190d33cd134c36bc3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c78cfa2f9504af6910dabc002fbe1da. This is attempt 1: this message will repeat every 5th retry.
I20250625 20:01:08.830106 2058 tablet_bootstrap.cc:492] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:08.830297 2055 raft_consensus.cc:2270] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.830354 1874 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet 53174bf6005e499c813b813b4a782c19 (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]) successfully deleted
I20250625 20:01:08.830857 2057 tablet_replica.cc:331] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:08.830964 2057 raft_consensus.cc:2241] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:08.831063 2057 raft_consensus.cc:2270] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.831277 1871 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet 7c78cfa2f9504af6910dabc002fbe1da (table default.table_53d5923f5fec4ca8b2fb669a4cf9395a [id=224a7f2d5a4d403fa2702fd59ccbf5c9]) successfully deleted
I20250625 20:01:08.831393 2057 ts_tablet_manager.cc:1905] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.831431 2058 tablet_bootstrap.cc:654] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.834437 2055 ts_tablet_manager.cc:1905] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.834704 2058 tablet_bootstrap.cc:492] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:08.834802 2058 ts_tablet_manager.cc:1397] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20250625 20:01:08.835268 2058 raft_consensus.cc:357] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.835323 2057 ts_tablet_manager.cc:1918] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.835387 2058 raft_consensus.cc:383] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.835445 2058 raft_consensus.cc:738] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.835582 2058 consensus_queue.cc:260] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.835959 2058 ts_tablet_manager.cc:1428] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 20:01:08.836097 2057 log.cc:1199] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/53174bf6005e499c813b813b4a782c19
I20250625 20:01:08.836339 2057 ts_tablet_manager.cc:1939] T 53174bf6005e499c813b813b4a782c19 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:08.838361 1871 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet 53174bf6005e499c813b813b4a782c19 (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]) successfully deleted
I20250625 20:01:08.838749 1827 raft_consensus.cc:491] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:08.838847 1827 raft_consensus.cc:513] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.839242 1827 leader_election.cc:290] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:08.840287 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac2af97f75484afab32084c274d890b3" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:08.840674 2057 tablet_replica.cc:331] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:08.841373 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac2af97f75484afab32084c274d890b3" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
W20250625 20:01:08.841827 1495 leader_election.cc:343] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): Illegal state: must be running to vote when last-logged opid is not known
W20250625 20:01:08.842025 1497 leader_election.cc:343] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Illegal state: must be running to vote when last-logged opid is not known
I20250625 20:01:08.842093 1497 leader_election.cc:304] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c73846c0665d4d53a274d053946107df; no voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08
I20250625 20:01:08.840811 2057 raft_consensus.cc:2241] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:08.842432 2057 raft_consensus.cc:2270] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.843094 2057 ts_tablet_manager.cc:1905] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.843914 1827 raft_consensus.cc:2747] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:01:08.845198 2057 ts_tablet_manager.cc:1918] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.845274 2057 log.cc:1199] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/9f41fcc4d1a14c3c94a3f20115926eec
I20250625 20:01:08.845518 2057 ts_tablet_manager.cc:1939] T 9f41fcc4d1a14c3c94a3f20115926eec P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:08.846273 2055 ts_tablet_manager.cc:1918] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.846263 1871 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet 9f41fcc4d1a14c3c94a3f20115926eec (table default.table_7b366e40d04d468caf65baf4c5b77004 [id=18d82881922e465cbcc90e87dfe4a5e1]) successfully deleted
I20250625 20:01:08.846369 2055 log.cc:1199] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/b36de85648bb4236a9320412cdb1cfdf
I20250625 20:01:08.846598 2055 ts_tablet_manager.cc:1939] T b36de85648bb4236a9320412cdb1cfdf P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:08.847206 2055 tablet_replica.cc:331] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:08.847302 1874 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet b36de85648bb4236a9320412cdb1cfdf (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]) successfully deleted
I20250625 20:01:08.847347 2055 raft_consensus.cc:2241] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:08.847512 2055 raft_consensus.cc:2270] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.848162 2061 tablet_bootstrap.cc:492] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:08.848264 2061 ts_tablet_manager.cc:1397] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.001s
I20250625 20:01:08.848722 2061 raft_consensus.cc:357] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.848832 2061 raft_consensus.cc:383] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.848860 2061 raft_consensus.cc:738] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.848989 2061 consensus_queue.cc:260] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.849259 2061 ts_tablet_manager.cc:1428] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250625 20:01:08.849440 2061 tablet_bootstrap.cc:492] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:08.850976 2061 tablet_bootstrap.cc:654] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.853864 2055 ts_tablet_manager.cc:1905] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.853874 2056 tablet_bootstrap.cc:492] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:08.854682 2056 ts_tablet_manager.cc:1397] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.000s
I20250625 20:01:08.855262 2056 raft_consensus.cc:357] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.856712 2061 tablet_bootstrap.cc:492] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:08.856801 2061 ts_tablet_manager.cc:1397] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20250625 20:01:08.856881 2056 raft_consensus.cc:383] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.856982 2056 raft_consensus.cc:738] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.857234 2056 consensus_queue.cc:260] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.857291 2061 raft_consensus.cc:357] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.857405 2061 raft_consensus.cc:383] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.857441 2061 raft_consensus.cc:738] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.857553 2061 consensus_queue.cc:260] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.856244 2055 ts_tablet_manager.cc:1918] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.858147 2055 log.cc:1199] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/3b7836328f054e2c95b12a8f0fe61b2b
I20250625 20:01:08.858392 2055 ts_tablet_manager.cc:1939] T 3b7836328f054e2c95b12a8f0fe61b2b P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:08.858515 1472 heartbeater.cc:499] Master 127.0.204.61:38005 was elected leader, sending a full tablet report...
I20250625 20:01:08.860113 2061 ts_tablet_manager.cc:1428] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250625 20:01:08.861789 1874 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet 3b7836328f054e2c95b12a8f0fe61b2b (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]) successfully deleted
I20250625 20:01:08.861851 2056 ts_tablet_manager.cc:1428] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250625 20:01:08.862100 2055 tablet_replica.cc:331] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:08.862286 2055 raft_consensus.cc:2241] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:08.862442 2055 raft_consensus.cc:2270] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.863847 2056 tablet_bootstrap.cc:492] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:08.864306 2055 ts_tablet_manager.cc:1905] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.864902 2056 tablet_bootstrap.cc:654] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.865908 2056 tablet_bootstrap.cc:492] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:08.866015 2056 ts_tablet_manager.cc:1397] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:08.866123 2055 ts_tablet_manager.cc:1918] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.866185 2055 log.cc:1199] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/936500cf83f84a1197231f4de02cd55b
I20250625 20:01:08.866662 2055 ts_tablet_manager.cc:1939] T 936500cf83f84a1197231f4de02cd55b P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:08.866804 2056 raft_consensus.cc:357] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.866906 2056 raft_consensus.cc:383] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.866953 2056 raft_consensus.cc:738] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.867084 2056 consensus_queue.cc:260] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.867231 2055 tablet_replica.cc:331] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:08.867318 2055 raft_consensus.cc:2241] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:08.867358 2056 ts_tablet_manager.cc:1428] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 20:01:08.867508 2055 raft_consensus.cc:2270] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.867928 2055 ts_tablet_manager.cc:1905] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.867522 2056 tablet_bootstrap.cc:492] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:08.871016 2056 tablet_bootstrap.cc:654] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.869907 2055 ts_tablet_manager.cc:1918] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.871590 2055 log.cc:1199] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/7c78cfa2f9504af6910dabc002fbe1da
I20250625 20:01:08.871866 2055 ts_tablet_manager.cc:1939] T 7c78cfa2f9504af6910dabc002fbe1da P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:08.872591 1874 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet 7c78cfa2f9504af6910dabc002fbe1da (table default.table_53d5923f5fec4ca8b2fb669a4cf9395a [id=224a7f2d5a4d403fa2702fd59ccbf5c9]) successfully deleted
I20250625 20:01:08.873085 2055 tablet_replica.cc:331] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:08.873214 2055 raft_consensus.cc:2241] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250625 20:01:08.873296 1874 catalog_manager.cc:4908] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): delete failed for tablet ac2af97f75484afab32084c274d890b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac2af97f75484afab32084c274d890b3 already in progress: creating tablet
I20250625 20:01:08.873329 2055 raft_consensus.cc:2270] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.873934 2055 ts_tablet_manager.cc:1905] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.876029 2055 ts_tablet_manager.cc:1918] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:08.876451 2056 tablet_bootstrap.cc:492] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:08.876543 2056 ts_tablet_manager.cc:1397] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250625 20:01:08.877039 2056 raft_consensus.cc:357] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.877156 2056 raft_consensus.cc:383] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.877192 2056 raft_consensus.cc:738] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.877326 2056 consensus_queue.cc:260] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.877704 2056 ts_tablet_manager.cc:1428] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:08.877884 2056 tablet_bootstrap.cc:492] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:08.878902 2056 tablet_bootstrap.cc:654] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.879410 2055 log.cc:1199] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/9f41fcc4d1a14c3c94a3f20115926eec
I20250625 20:01:08.879667 2055 ts_tablet_manager.cc:1939] T 9f41fcc4d1a14c3c94a3f20115926eec P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:08.880364 1874 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet 9f41fcc4d1a14c3c94a3f20115926eec (table default.table_7b366e40d04d468caf65baf4c5b77004 [id=18d82881922e465cbcc90e87dfe4a5e1]) successfully deleted
I20250625 20:01:08.881249 2056 tablet_bootstrap.cc:492] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:08.881340 2056 ts_tablet_manager.cc:1397] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 20:01:08.881789 2056 raft_consensus.cc:357] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.881896 2056 raft_consensus.cc:383] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.881932 2056 raft_consensus.cc:738] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.882048 2056 consensus_queue.cc:260] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.882285 2056 ts_tablet_manager.cc:1428] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:08.888562 1091 catalog_manager.cc:2149] created HMS catalog entry for table [id=65222a2a9f7e44099a490760d1f51ce4]
W20250625 20:01:08.889117 1091 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 5852fb53503b438c92f294e0c44c916d is not leader of this config: current role FOLLOWER
I20250625 20:01:08.902895 1827 raft_consensus.cc:491] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:08.903030 1827 raft_consensus.cc:513] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.903440 1827 leader_election.cc:290] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:08.904150 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07686deb2e184c9fbae06fef5365d0d5" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:08.904297 1424 raft_consensus.cc:2466] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:08.904289 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07686deb2e184c9fbae06fef5365d0d5" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:08.904438 1701 raft_consensus.cc:2466] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:08.904791 1497 leader_election.cc:304] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:08.905038 1827 raft_consensus.cc:2802] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:08.905087 1827 raft_consensus.cc:491] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:08.905121 1827 raft_consensus.cc:3058] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:08.906179 1827 raft_consensus.cc:513] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.906553 1827 leader_election.cc:290] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:08.906966 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07686deb2e184c9fbae06fef5365d0d5" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:08.907091 1701 raft_consensus.cc:3058] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:08.907115 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07686deb2e184c9fbae06fef5365d0d5" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:08.907236 1424 raft_consensus.cc:3058] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:08.908159 1701 raft_consensus.cc:2466] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:08.908175 1424 raft_consensus.cc:2466] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:08.908488 1495 leader_election.cc:304] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:08.908687 1827 raft_consensus.cc:2802] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:08.908762 1827 raft_consensus.cc:695] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Becoming Leader. State: Replica: c73846c0665d4d53a274d053946107df, State: Running, Role: LEADER
I20250625 20:01:08.908906 1827 consensus_queue.cc:237] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.910619 1879 catalog_manager.cc:2149] created HMS catalog entry for table [id=a054c42074394f1db21dfcf2c55d99b5]
I20250625 20:01:08.920608 1675 tablet_service.cc:1468] Processing CreateTablet for tablet 78631bcca66d4b16b344e57f8a1cc291 (DEFAULT_TABLE table=default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.920608 1537 tablet_service.cc:1468] Processing CreateTablet for tablet 78631bcca66d4b16b344e57f8a1cc291 (DEFAULT_TABLE table=default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.920946 1675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78631bcca66d4b16b344e57f8a1cc291. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:08.920946 1537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78631bcca66d4b16b344e57f8a1cc291. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:08.921392 1399 tablet_service.cc:1468] Processing CreateTablet for tablet 78631bcca66d4b16b344e57f8a1cc291 (DEFAULT_TABLE table=default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:08.921720 1399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78631bcca66d4b16b344e57f8a1cc291. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:08.923178 2061 tablet_bootstrap.cc:492] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:08.924015 2056 tablet_bootstrap.cc:492] T 78631bcca66d4b16b344e57f8a1cc291 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:08.924243 2061 tablet_bootstrap.cc:654] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.924978 2056 tablet_bootstrap.cc:654] T 78631bcca66d4b16b344e57f8a1cc291 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.925105 2058 tablet_bootstrap.cc:492] T 78631bcca66d4b16b344e57f8a1cc291 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:08.926005 2058 tablet_bootstrap.cc:654] T 78631bcca66d4b16b344e57f8a1cc291 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:08.926231 1820 raft_consensus.cc:491] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:08.926321 1820 raft_consensus.cc:513] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.926731 1820 leader_election.cc:290] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:08.927215 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5d6a94383be4a1dba614c0db1b88003" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:08.927258 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5d6a94383be4a1dba614c0db1b88003" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:08.927348 1701 raft_consensus.cc:2466] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:08.927388 1560 raft_consensus.cc:2466] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:08.927662 1358 leader_election.cc:304] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:08.927907 1820 raft_consensus.cc:2802] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:08.927971 1820 raft_consensus.cc:491] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:08.928054 1820 raft_consensus.cc:3058] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:08.928241 1879 catalog_manager.cc:5674] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561) does not have the latest schema for tablet b78b6f29890140de958cebed9868b303 (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]). Expected version 1 got 0
I20250625 20:01:08.929055 1820 raft_consensus.cc:513] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.929451 1820 leader_election.cc:290] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:08.929894 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5d6a94383be4a1dba614c0db1b88003" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:08.930040 1701 raft_consensus.cc:3058] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:08.930047 1879 catalog_manager.cc:5674] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255) does not have the latest schema for tablet b78b6f29890140de958cebed9868b303 (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]). Expected version 1 got 0
I20250625 20:01:08.930197 1879 catalog_manager.cc:5674] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255) does not have the latest schema for tablet 07686deb2e184c9fbae06fef5365d0d5 (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]). Expected version 1 got 0
I20250625 20:01:08.930303 1879 catalog_manager.cc:5674] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255) does not have the latest schema for tablet d5d6a94383be4a1dba614c0db1b88003 (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]). Expected version 1 got 0
I20250625 20:01:08.931061 1701 raft_consensus.cc:2466] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:08.931535 1358 leader_election.cc:304] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:08.931947 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5d6a94383be4a1dba614c0db1b88003" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:08.932094 1560 raft_consensus.cc:3058] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:08.933161 1560 raft_consensus.cc:2466] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:08.933609 1820 raft_consensus.cc:2802] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:08.933709 1820 raft_consensus.cc:695] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:08.933851 1820 consensus_queue.cc:237] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.934110 1674 tablet_service.cc:1515] Processing DeleteTablet for tablet ac2af97f75484afab32084c274d890b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:43128
I20250625 20:01:08.934311 2060 tablet_replica.cc:331] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:08.934448 2060 raft_consensus.cc:2241] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:08.934561 2060 raft_consensus.cc:2270] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.934893 1675 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f41fcc4d1a14c3c94a3f20115926eec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:43128
I20250625 20:01:08.935125 2060 ts_tablet_manager.cc:1905] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.935750 1843 raft_consensus.cc:491] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:08.935851 1843 raft_consensus.cc:513] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.936240 1843 leader_election.cc:290] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:08.937382 2060 ts_tablet_manager.cc:1918] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 20:01:08.937465 2060 log.cc:1199] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/ac2af97f75484afab32084c274d890b3
I20250625 20:01:08.937862 2060 ts_tablet_manager.cc:1939] T ac2af97f75484afab32084c274d890b3 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:08.938849 1879 catalog_manager.cc:5674] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759) does not have the latest schema for tablet b78b6f29890140de958cebed9868b303 (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]). Expected version 1 got 0
I20250625 20:01:08.939081 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b78b6f29890140de958cebed9868b303" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:08.939231 1424 raft_consensus.cc:2466] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:08.940117 1879 catalog_manager.cc:5674] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759) does not have the latest schema for tablet 07686deb2e184c9fbae06fef5365d0d5 (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]). Expected version 1 got 0
I20250625 20:01:08.940308 1879 catalog_manager.cc:5674] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759) does not have the latest schema for tablet d5d6a94383be4a1dba614c0db1b88003 (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]). Expected version 1 got 0
I20250625 20:01:08.941323 1637 leader_election.cc:304] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:08.941540 1843 raft_consensus.cc:2802] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:08.941603 1843 raft_consensus.cc:491] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:08.941650 1843 raft_consensus.cc:3058] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:08.941890 1399 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c78cfa2f9504af6910dabc002fbe1da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:46136
W20250625 20:01:08.942487 1874 catalog_manager.cc:4895] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): delete failed for tablet 7c78cfa2f9504af6910dabc002fbe1da because the tablet was not found. No further retry: Not found: Tablet not found: 7c78cfa2f9504af6910dabc002fbe1da
I20250625 20:01:08.942677 1843 raft_consensus.cc:513] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.943097 1843 leader_election.cc:290] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:08.943233 1398 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b7836328f054e2c95b12a8f0fe61b2b with delete_type TABLET_DATA_DELETED (replaced by tablet b78b6f29890140de958cebed9868b303) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:08.943408 1397 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f41fcc4d1a14c3c94a3f20115926eec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:08.943526 1400 tablet_service.cc:1515] Processing DeleteTablet for tablet ac2af97f75484afab32084c274d890b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:08.943550 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b78b6f29890140de958cebed9868b303" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:08.943684 1561 raft_consensus.cc:3058] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:08.943776 2055 tablet_replica.cc:331] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:08.943797 1399 tablet_service.cc:1515] Processing DeleteTablet for tablet 936500cf83f84a1197231f4de02cd55b with delete_type TABLET_DATA_DELETED (replaced by tablet ac2af97f75484afab32084c274d890b3) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:08.943895 2055 raft_consensus.cc:2241] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:08.944007 2055 raft_consensus.cc:2270] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 20:01:08.944195 1874 catalog_manager.cc:4895] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): delete failed for tablet 3b7836328f054e2c95b12a8f0fe61b2b because the tablet was not found. No further retry: Not found: Tablet not found: 3b7836328f054e2c95b12a8f0fe61b2b
W20250625 20:01:08.944360 1874 catalog_manager.cc:4895] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): delete failed for tablet 9f41fcc4d1a14c3c94a3f20115926eec because the tablet was not found. No further retry: Not found: Tablet not found: 9f41fcc4d1a14c3c94a3f20115926eec
I20250625 20:01:08.944826 2055 ts_tablet_manager.cc:1905] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.944818 1561 raft_consensus.cc:2466] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:08.945191 1634 leader_election.cc:304] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:08.946154 1879 catalog_manager.cc:5674] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255) does not have the latest schema for tablet b78b6f29890140de958cebed9868b303 (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]). Expected version 1 got 0
I20250625 20:01:08.946341 1879 catalog_manager.cc:5674] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255) does not have the latest schema for tablet 07686deb2e184c9fbae06fef5365d0d5 (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]). Expected version 1 got 0
I20250625 20:01:08.946450 1879 catalog_manager.cc:5674] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255) does not have the latest schema for tablet d5d6a94383be4a1dba614c0db1b88003 (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]). Expected version 1 got 0
I20250625 20:01:08.946892 1872 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet ac2af97f75484afab32084c274d890b3 (table default.table_4c6973f983694e48873d7009b44c2d4d [id=af2791dc33fb464db4a937aebe316482]) successfully deleted
W20250625 20:01:08.947062 1872 catalog_manager.cc:4895] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): delete failed for tablet 9f41fcc4d1a14c3c94a3f20115926eec because the tablet was not found. No further retry: Not found: Tablet not found: 9f41fcc4d1a14c3c94a3f20115926eec
I20250625 20:01:08.947814 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b78b6f29890140de958cebed9868b303" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:08.947933 1424 raft_consensus.cc:3058] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:08.948048 1675 tablet_service.cc:1515] Processing DeleteTablet for tablet ac2af97f75484afab32084c274d890b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:43128
I20250625 20:01:08.948256 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b78b6f29890140de958cebed9868b303" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:08.948369 1560 raft_consensus.cc:2374] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1. Re-sending same reply.
I20250625 20:01:08.948766 1843 raft_consensus.cc:2802] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:08.948810 1424 raft_consensus.cc:2466] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:08.948863 1843 raft_consensus.cc:695] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
I20250625 20:01:08.949010 1843 consensus_queue.cc:237] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:08.949400 1879 catalog_manager.cc:5674] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561) does not have the latest schema for tablet b78b6f29890140de958cebed9868b303 (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]). Expected version 1 got 0
I20250625 20:01:08.949479 2055 ts_tablet_manager.cc:1918] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 20:01:08.949553 2055 log.cc:1199] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/ac2af97f75484afab32084c274d890b3
I20250625 20:01:08.949762 2055 ts_tablet_manager.cc:1939] T ac2af97f75484afab32084c274d890b3 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:08.949674 1879 catalog_manager.cc:5582] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df reported cstate change: term changed from 0 to 1, leader changed from <none> to c73846c0665d4d53a274d053946107df (127.0.204.2). New cstate: current_term: 1 leader_uuid: "c73846c0665d4d53a274d053946107df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } health_report { overall_health: HEALTHY } } }
I20250625 20:01:08.950031 1879 catalog_manager.cc:5674] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561) does not have the latest schema for tablet 07686deb2e184c9fbae06fef5365d0d5 (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]). Expected version 1 got 0
I20250625 20:01:08.950186 1879 catalog_manager.cc:5674] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561) does not have the latest schema for tablet d5d6a94383be4a1dba614c0db1b88003 (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]). Expected version 1 got 0
I20250625 20:01:08.951480 1874 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet ac2af97f75484afab32084c274d890b3 (table default.table_4c6973f983694e48873d7009b44c2d4d [id=af2791dc33fb464db4a937aebe316482]) successfully deleted
W20250625 20:01:08.951640 1874 catalog_manager.cc:4895] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): delete failed for tablet 936500cf83f84a1197231f4de02cd55b because the tablet was not found. No further retry: Not found: Tablet not found: 936500cf83f84a1197231f4de02cd55b
W20250625 20:01:08.952436 1872 catalog_manager.cc:4895] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): delete failed for tablet ac2af97f75484afab32084c274d890b3 because the tablet was not found. No further retry: Not found: Tablet not found: ac2af97f75484afab32084c274d890b3
I20250625 20:01:08.953442 2056 tablet_bootstrap.cc:492] T 78631bcca66d4b16b344e57f8a1cc291 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:08.953554 2056 ts_tablet_manager.cc:1397] T 78631bcca66d4b16b344e57f8a1cc291 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250625 20:01:08.953848 2058 tablet_bootstrap.cc:492] T 78631bcca66d4b16b344e57f8a1cc291 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:08.953938 2058 ts_tablet_manager.cc:1397] T 78631bcca66d4b16b344e57f8a1cc291 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.002s
I20250625 20:01:08.954075 2056 raft_consensus.cc:357] T 78631bcca66d4b16b344e57f8a1cc291 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.954175 2056 raft_consensus.cc:383] T 78631bcca66d4b16b344e57f8a1cc291 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.954203 2056 raft_consensus.cc:738] T 78631bcca66d4b16b344e57f8a1cc291 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.954322 2056 consensus_queue.cc:260] T 78631bcca66d4b16b344e57f8a1cc291 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.954442 2058 raft_consensus.cc:357] T 78631bcca66d4b16b344e57f8a1cc291 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.954543 2058 raft_consensus.cc:383] T 78631bcca66d4b16b344e57f8a1cc291 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.954574 2058 raft_consensus.cc:738] T 78631bcca66d4b16b344e57f8a1cc291 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.954603 2056 ts_tablet_manager.cc:1428] T 78631bcca66d4b16b344e57f8a1cc291 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:08.954686 2058 consensus_queue.cc:260] T 78631bcca66d4b16b344e57f8a1cc291 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.954983 2058 ts_tablet_manager.cc:1428] T 78631bcca66d4b16b344e57f8a1cc291 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 20:01:08.960216 1537 tablet_service.cc:1515] Processing DeleteTablet for tablet ac2af97f75484afab32084c274d890b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:57380
I20250625 20:01:08.960420 2057 tablet_replica.cc:331] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:08.960549 2057 raft_consensus.cc:2241] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:08.960680 2057 raft_consensus.cc:2270] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:08.961220 2057 ts_tablet_manager.cc:1905] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:08.962642 2061 tablet_bootstrap.cc:492] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:08.962738 2061 ts_tablet_manager.cc:1397] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.000s
I20250625 20:01:08.963258 2061 raft_consensus.cc:357] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.963361 2061 raft_consensus.cc:383] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:08.963387 2061 raft_consensus.cc:738] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:08.963498 2057 ts_tablet_manager.cc:1918] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 20:01:08.963495 2061 consensus_queue.cc:260] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.963564 2057 log.cc:1199] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/ac2af97f75484afab32084c274d890b3
I20250625 20:01:08.963797 2061 ts_tablet_manager.cc:1428] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:08.965677 2057 ts_tablet_manager.cc:1939] T ac2af97f75484afab32084c274d890b3 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:08.966004 1879 catalog_manager.cc:5674] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759) does not have the latest schema for tablet b78b6f29890140de958cebed9868b303 (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]). Expected version 1 got 0
I20250625 20:01:08.966216 1879 catalog_manager.cc:5582] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1). New cstate: current_term: 1 leader_uuid: "27aa5b53b16747f9a72485aaf2417b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } health_report { overall_health: UNKNOWN } } }
I20250625 20:01:08.966372 1871 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet ac2af97f75484afab32084c274d890b3 (table default.table_4c6973f983694e48873d7009b44c2d4d [id=af2791dc33fb464db4a937aebe316482]) successfully deleted
I20250625 20:01:08.966516 1879 catalog_manager.cc:5674] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759) does not have the latest schema for tablet d5d6a94383be4a1dba614c0db1b88003 (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]). Expected version 1 got 0
I20250625 20:01:08.966670 1879 catalog_manager.cc:5674] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759) does not have the latest schema for tablet 07686deb2e184c9fbae06fef5365d0d5 (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]). Expected version 1 got 0
I20250625 20:01:08.969731 1843 raft_consensus.cc:491] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:08.969867 1843 raft_consensus.cc:513] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.970253 1843 leader_election.cc:290] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:08.970670 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78631bcca66d4b16b344e57f8a1cc291" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:08.970816 1560 raft_consensus.cc:2466] T 78631bcca66d4b16b344e57f8a1cc291 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:08.972002 1634 leader_election.cc:304] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:08.972369 1843 raft_consensus.cc:2802] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:08.972430 1843 raft_consensus.cc:491] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:08.972466 1843 raft_consensus.cc:3058] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:08.972481 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78631bcca66d4b16b344e57f8a1cc291" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:08.972611 1424 raft_consensus.cc:2466] T 78631bcca66d4b16b344e57f8a1cc291 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:08.973534 1843 raft_consensus.cc:513] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.973891 1843 leader_election.cc:290] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:08.974277 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78631bcca66d4b16b344e57f8a1cc291" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:08.974404 1424 raft_consensus.cc:3058] T 78631bcca66d4b16b344e57f8a1cc291 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:08.974416 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78631bcca66d4b16b344e57f8a1cc291" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:08.974534 1560 raft_consensus.cc:3058] T 78631bcca66d4b16b344e57f8a1cc291 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:08.975373 1424 raft_consensus.cc:2466] T 78631bcca66d4b16b344e57f8a1cc291 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:08.975418 1560 raft_consensus.cc:2466] T 78631bcca66d4b16b344e57f8a1cc291 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:08.975646 1399 tablet_service.cc:1515] Processing DeleteTablet for tablet ac2af97f75484afab32084c274d890b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:01:08 UTC) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:08.975912 1637 leader_election.cc:304] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:08.976158 1843 raft_consensus.cc:2802] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:08.976238 1843 raft_consensus.cc:695] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
W20250625 20:01:08.976316 1874 catalog_manager.cc:4895] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): delete failed for tablet ac2af97f75484afab32084c274d890b3 because the tablet was not found. No further retry: Not found: Tablet not found: ac2af97f75484afab32084c274d890b3
I20250625 20:01:08.976359 1843 consensus_queue.cc:237] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:08.976572 1879 catalog_manager.cc:5582] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3). New cstate: current_term: 1 leader_uuid: "0ed894f2ff26414190d33cd134c36bc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } health_report { overall_health: UNKNOWN } } }
I20250625 20:01:08.976764 1879 catalog_manager.cc:5674] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255) does not have the latest schema for tablet b78b6f29890140de958cebed9868b303 (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]). Expected version 1 got 0
I20250625 20:01:08.983114 1879 catalog_manager.cc:5582] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3). New cstate: current_term: 1 leader_uuid: "0ed894f2ff26414190d33cd134c36bc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } health_report { overall_health: HEALTHY } } }
I20250625 20:01:08.990079 1879 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49704:
table { table_name: "default.Table_030bc68e40784cf2a6bd261221a0f738" } modify_external_catalogs: true
I20250625 20:01:08.990309 1879 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49704:
table { table_name: "default.Table_030bc68e40784cf2a6bd261221a0f738" } modify_external_catalogs: true
I20250625 20:01:09.006584 1889 catalog_manager.cc:2149] created HMS catalog entry for table [id=aee290f61c8f4d7b93d4cd7192eb9277]
I20250625 20:01:09.011541 1889 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49704:
table { table_name: "default.Table_d0f8eb84406f4d758e34b162746daa38" } new_table_name: "default.Table_a78f24eb6417409489ec4469005095b4" modify_external_catalogs: true
I20250625 20:01:09.019738 1399 tablet_service.cc:1468] Processing CreateTablet for tablet cdb2a543898642d3abc3afa850178120 (DEFAULT_TABLE table=default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.019980 1537 tablet_service.cc:1468] Processing CreateTablet for tablet cdb2a543898642d3abc3afa850178120 (DEFAULT_TABLE table=default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.020103 1399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb2a543898642d3abc3afa850178120. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.020258 1537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb2a543898642d3abc3afa850178120. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.021073 1087 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=81aee385d1404c82aaeebee220cb9e68]
I20250625 20:01:09.021262 1087 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41858 (request call id 48) took 1832 ms. Trace:
I20250625 20:01:09.021317 1087 rpcz_store.cc:276] 0625 20:01:07.189107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:07.319732 (+130625us) service_pool.cc:225] Handling call
0625 20:01:07.804028 (+484296us) catalog_manager.cc:2076] Acquired catalog manager lock
0625 20:01:07.804311 (+ 283us) catalog_manager.cc:2122] Created new table and tablet info
0625 20:01:08.573539 (+769228us) catalog_manager.cc:2148] Created new table in HMS catalog
0625 20:01:08.677773 (+104234us) catalog_manager.cc:2156] Rolling back HMS table creation
0625 20:01:09.021250 (+343477us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 20:01:08.574329 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 20:01:08.574424 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0625 20:01:08.574428 (+ 4us) write_op.cc:435] Acquired schema lock
0625 20:01:08.574430 (+ 2us) tablet.cc:660] Decoding operations
0625 20:01:08.574502 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0625 20:01:08.574513 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0625 20:01:08.574516 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0625 20:01:08.574574 (+ 58us) tablet.cc:699] Row locks acquired
0625 20:01:08.574575 (+ 1us) write_op.cc:260] PREPARE: finished
0625 20:01:08.574604 (+ 29us) write_op.cc:270] Start()
0625 20:01:08.574630 (+ 26us) write_op.cc:276] Timestamp: P: 1750881668574578 usec, L: 0
0625 20:01:08.574632 (+ 2us) op_driver.cc:348] REPLICATION: starting
0625 20:01:08.574782 (+ 150us) log.cc:844] Serialized 432 byte log entry
0625 20:01:08.677007 (+102225us) op_driver.cc:464] REPLICATION: finished
0625 20:01:08.677028 (+ 21us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0625 20:01:08.677034 (+ 6us) write_op.cc:489] Releasing partition, row and schema locks
0625 20:01:08.677058 (+ 24us) write_op.cc:454] Released schema lock
0625 20:01:08.677064 (+ 6us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":939060,"Hive Metastore.run_cpu_time_us":404,"Hive Metastore.run_wall_time_us":173146,"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":277,"prepare.run_cpu_time_us":688,"prepare.run_wall_time_us":959,"raft.queue_time_us":8644,"raft.run_cpu_time_us":491,"raft.run_wall_time_us":3488,"replication_time_us":102324,"thread_start_us":233,"threads_started":3}]]}
I20250625 20:01:09.022655 2058 tablet_bootstrap.cc:492] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:09.022734 2056 tablet_bootstrap.cc:492] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:09.023772 2056 tablet_bootstrap.cc:654] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.023818 2058 tablet_bootstrap.cc:654] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.025175 1675 tablet_service.cc:1468] Processing CreateTablet for tablet cdb2a543898642d3abc3afa850178120 (DEFAULT_TABLE table=default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.025452 1675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb2a543898642d3abc3afa850178120. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.027567 2056 tablet_bootstrap.cc:492] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:09.027695 2056 ts_tablet_manager.cc:1397] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20250625 20:01:09.028293 2056 raft_consensus.cc:357] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.028417 2056 raft_consensus.cc:383] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.028453 2056 raft_consensus.cc:738] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.028584 2056 consensus_queue.cc:260] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.028883 2056 ts_tablet_manager.cc:1428] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 20:01:09.031025 2058 tablet_bootstrap.cc:492] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:09.031152 2058 ts_tablet_manager.cc:1397] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20250625 20:01:09.032305 2061 tablet_bootstrap.cc:492] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:09.033378 2061 tablet_bootstrap.cc:654] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.034605 2061 tablet_bootstrap.cc:492] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:09.034704 2061 ts_tablet_manager.cc:1397] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250625 20:01:09.035192 2061 raft_consensus.cc:357] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.035305 2061 raft_consensus.cc:383] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.035346 2061 raft_consensus.cc:738] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.035480 2061 consensus_queue.cc:260] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.031638 2058 raft_consensus.cc:357] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.035683 2058 raft_consensus.cc:383] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.035750 2058 raft_consensus.cc:738] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.036052 2061 ts_tablet_manager.cc:1428] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:09.036195 2058 consensus_queue.cc:260] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.036494 2058 ts_tablet_manager.cc:1428] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250625 20:01:09.060012 1560 raft_consensus.cc:1273] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Refusing update from remote peer 27aa5b53b16747f9a72485aaf2417b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 20:01:09.060109 1701 raft_consensus.cc:1273] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Refusing update from remote peer 27aa5b53b16747f9a72485aaf2417b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 20:01:09.060473 1820 consensus_queue.cc:1035] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.060623 1850 consensus_queue.cc:1035] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.066563 1978 mvcc.cc:204] Tried to move back new op lower bound from 7171611316464771072 to 7171611315954470912. Current Snapshot: MvccSnapshot[applied={T|T < 7171611316464771072}]
I20250625 20:01:09.068460 2087 tablet.cc:1722] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 20:01:09.068553 1977 mvcc.cc:204] Tried to move back new op lower bound from 7171611316464771072 to 7171611315954470912. Current Snapshot: MvccSnapshot[applied={T|T < 7171611316464771072}]
I20250625 20:01:09.071300 2086 tablet.cc:1722] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 20:01:09.072701 1976 mvcc.cc:204] Tried to move back new op lower bound from 7171611316464771072 to 7171611315954470912. Current Snapshot: MvccSnapshot[applied={T|T < 7171611316464771072}]
I20250625 20:01:09.078215 2088 tablet.cc:1722] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 20:01:09.081073 1701 raft_consensus.cc:1273] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Refusing update from remote peer c73846c0665d4d53a274d053946107df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 20:01:09.081429 1424 raft_consensus.cc:1273] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Refusing update from remote peer c73846c0665d4d53a274d053946107df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 20:01:09.081631 1935 consensus_queue.cc:1035] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.082118 1935 consensus_queue.cc:1035] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.090690 2086 tablet.cc:1722] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 20:01:09.091184 2087 tablet.cc:1722] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 20:01:09.095501 2088 tablet.cc:1722] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 20:01:09.104574 1560 raft_consensus.cc:1273] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Refusing update from remote peer 0ed894f2ff26414190d33cd134c36bc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 20:01:09.104574 1424 raft_consensus.cc:1273] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Refusing update from remote peer 0ed894f2ff26414190d33cd134c36bc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 20:01:09.104974 1829 consensus_queue.cc:1035] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.105276 1843 consensus_queue.cc:1035] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.112879 2088 tablet.cc:1722] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 20:01:09.114184 2087 tablet.cc:1722] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 20:01:09.121645 2086 tablet.cc:1722] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 20:01:09.144863 1883 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]
I20250625 20:01:09.146508 1091 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=65222a2a9f7e44099a490760d1f51ce4]
I20250625 20:01:09.146785 1091 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41858 (request call id 52) took 1834 ms. Trace:
I20250625 20:01:09.146870 1091 rpcz_store.cc:276] 0625 20:01:07.312602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 20:01:08.047046 (+734444us) service_pool.cc:225] Handling call
0625 20:01:08.646023 (+598977us) catalog_manager.cc:2076] Acquired catalog manager lock
0625 20:01:08.646446 (+ 423us) catalog_manager.cc:2122] Created new table and tablet info
0625 20:01:08.888551 (+242105us) catalog_manager.cc:2148] Created new table in HMS catalog
0625 20:01:08.889069 (+ 518us) write_op.cc:489] Releasing partition, row and schema locks
0625 20:01:08.889071 (+ 2us) write_op.cc:454] Released schema lock
0625 20:01:08.889202 (+ 131us) catalog_manager.cc:2156] Rolling back HMS table creation
0625 20:01:09.146758 (+257556us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 20:01:08.889022 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 5852fb53503b438c92f294e0c44c916d is not leader of this config: current role FOLLOWER)
0625 20:01:08.889030 (+ 8us) write_op.cc:489] Releasing partition, row and schema locks
0625 20:01:08.889035 (+ 5us) write_op.cc:454] Released schema lock
0625 20:01:08.889038 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":259121,"Hive Metastore.run_cpu_time_us":367,"Hive Metastore.run_wall_time_us":240021,"child_traces":[["op",{}]]}
I20250625 20:01:09.164721 1882 catalog_manager.cc:2149] created HMS catalog entry for table [id=de00fa1efd2041b29fa3e9d329f6043f]
I20250625 20:01:09.176584 1537 tablet_service.cc:1468] Processing CreateTablet for tablet 2a388d1449bf455dbde985974c23e43f (DEFAULT_TABLE table=default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.176908 1537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a388d1449bf455dbde985974c23e43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.178567 2058 tablet_bootstrap.cc:492] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:09.179450 1675 tablet_service.cc:1468] Processing CreateTablet for tablet 2a388d1449bf455dbde985974c23e43f (DEFAULT_TABLE table=default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.179701 2058 tablet_bootstrap.cc:654] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.179759 1675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a388d1449bf455dbde985974c23e43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.180882 2058 tablet_bootstrap.cc:492] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:09.180964 2058 ts_tablet_manager.cc:1397] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250625 20:01:09.181432 2058 raft_consensus.cc:357] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.181538 2058 raft_consensus.cc:383] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.181569 2058 raft_consensus.cc:738] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.181686 2058 consensus_queue.cc:260] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.181942 2058 ts_tablet_manager.cc:1428] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 20:01:09.182193 2061 tablet_bootstrap.cc:492] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:09.183172 2061 tablet_bootstrap.cc:654] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.185081 1399 tablet_service.cc:1468] Processing CreateTablet for tablet 2a388d1449bf455dbde985974c23e43f (DEFAULT_TABLE table=default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.185410 1399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a388d1449bf455dbde985974c23e43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.186849 1882 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49704:
table { table_name: "default.Table_8302314b76364eadb44f02cc6d58a589" } new_table_name: "default.Table_2fcd1dbf63224757b5e1a9bad86e6a77" modify_external_catalogs: true
I20250625 20:01:09.187862 2056 tablet_bootstrap.cc:492] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:09.188172 2061 tablet_bootstrap.cc:492] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:09.188257 2061 ts_tablet_manager.cc:1397] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250625 20:01:09.188827 2061 raft_consensus.cc:357] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.188922 2061 raft_consensus.cc:383] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.188951 2061 raft_consensus.cc:738] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.188961 2056 tablet_bootstrap.cc:654] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.189066 2061 consensus_queue.cc:260] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.189314 2061 ts_tablet_manager.cc:1428] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:09.193724 2056 tablet_bootstrap.cc:492] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:09.193821 2056 ts_tablet_manager.cc:1397] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20250625 20:01:09.194334 2056 raft_consensus.cc:357] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.194425 2056 raft_consensus.cc:383] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.194454 2056 raft_consensus.cc:738] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.194574 2056 consensus_queue.cc:260] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.194801 2056 ts_tablet_manager.cc:1428] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 20:01:09.195436 1884 catalog_manager.cc:2149] created HMS catalog entry for table [id=66512a85e5d34065afe0c5ca62599bc4]
I20250625 20:01:09.209434 1537 tablet_service.cc:1468] Processing CreateTablet for tablet 6c8e829c96474e94a8de9e7e32b48350 (DEFAULT_TABLE table=default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.209761 1537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8e829c96474e94a8de9e7e32b48350. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.210444 1399 tablet_service.cc:1468] Processing CreateTablet for tablet 6c8e829c96474e94a8de9e7e32b48350 (DEFAULT_TABLE table=default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.210777 1399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8e829c96474e94a8de9e7e32b48350. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.212082 2058 tablet_bootstrap.cc:492] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:09.213074 2056 tablet_bootstrap.cc:492] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:09.213107 2058 tablet_bootstrap.cc:654] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.214128 2056 tablet_bootstrap.cc:654] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.214674 1675 tablet_service.cc:1468] Processing CreateTablet for tablet 6c8e829c96474e94a8de9e7e32b48350 (DEFAULT_TABLE table=default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.214951 1675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8e829c96474e94a8de9e7e32b48350. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.217483 2061 tablet_bootstrap.cc:492] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:09.218456 2061 tablet_bootstrap.cc:654] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.220068 1884 catalog_manager.cc:6127] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0] alter complete (version 1)
I20250625 20:01:09.222741 2056 tablet_bootstrap.cc:492] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:09.222877 2056 ts_tablet_manager.cc:1397] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20250625 20:01:09.223496 2056 raft_consensus.cc:357] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.223628 2056 raft_consensus.cc:383] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.223665 2056 raft_consensus.cc:738] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.223842 2056 consensus_queue.cc:260] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.224184 2056 ts_tablet_manager.cc:1428] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250625 20:01:09.225922 1884 catalog_manager.cc:6127] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1] alter complete (version 1)
I20250625 20:01:09.228374 2061 tablet_bootstrap.cc:492] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:09.228485 2061 ts_tablet_manager.cc:1397] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250625 20:01:09.228533 2058 tablet_bootstrap.cc:492] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:09.228622 2058 ts_tablet_manager.cc:1397] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20250625 20:01:09.229032 2061 raft_consensus.cc:357] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.229125 2061 raft_consensus.cc:383] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.229148 2061 raft_consensus.cc:738] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.229118 2058 raft_consensus.cc:357] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.229225 2058 raft_consensus.cc:383] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.229259 2058 raft_consensus.cc:738] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.229250 2061 consensus_queue.cc:260] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.229383 2058 consensus_queue.cc:260] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.229508 2061 ts_tablet_manager.cc:1428] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 20:01:09.229965 1991 raft_consensus.cc:491] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:09.230070 1991 raft_consensus.cc:513] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.230491 1991 leader_election.cc:290] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:09.231843 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c8e829c96474e94a8de9e7e32b48350" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:09.231956 1701 raft_consensus.cc:2466] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:09.232267 1358 leader_election.cc:304] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:09.232441 1991 raft_consensus.cc:2802] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:09.232426 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c8e829c96474e94a8de9e7e32b48350" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:09.232489 1991 raft_consensus.cc:491] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:09.232530 1991 raft_consensus.cc:3058] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.232532 1560 raft_consensus.cc:2466] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:09.233632 1991 raft_consensus.cc:513] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.234010 1991 leader_election.cc:290] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:09.235057 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c8e829c96474e94a8de9e7e32b48350" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:09.235173 1560 raft_consensus.cc:3058] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.236090 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c8e829c96474e94a8de9e7e32b48350" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:09.236195 1701 raft_consensus.cc:3058] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.236222 1560 raft_consensus.cc:2466] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:09.236657 1357 leader_election.cc:304] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:09.236843 1991 raft_consensus.cc:2802] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:09.236918 1991 raft_consensus.cc:695] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:09.237048 1991 consensus_queue.cc:237] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.237224 1701 raft_consensus.cc:2466] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:09.244760 1884 catalog_manager.cc:6127] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809] alter complete (version 1)
I20250625 20:01:09.249986 1884 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49704:
table { table_name: "default.Table_fd28d4c5b48245c382a334b98a3066df" } modify_external_catalogs: true
I20250625 20:01:09.250159 1884 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49704:
table { table_name: "default.Table_fd28d4c5b48245c382a334b98a3066df" } modify_external_catalogs: true
I20250625 20:01:09.251444 2058 ts_tablet_manager.cc:1428] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20250625 20:01:09.261385 1887 catalog_manager.cc:3460] renamed table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583] in HMS: new name default.table_ad0e628a3e004a9da4a42cb7d6051e3e
I20250625 20:01:09.285646 1885 catalog_manager.cc:2149] created HMS catalog entry for table [id=0f4987c9c14c46a89443ce1b66ceaaaa]
I20250625 20:01:09.300789 2093 raft_consensus.cc:491] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:09.300933 2093 raft_consensus.cc:513] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.301445 2093 leader_election.cc:290] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:09.301976 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a388d1449bf455dbde985974c23e43f" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:09.302142 1424 raft_consensus.cc:2466] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:09.303121 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a388d1449bf455dbde985974c23e43f" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:09.303203 1537 tablet_service.cc:1468] Processing CreateTablet for tablet c9a69c50e5f14364825e048fd19d4409 (DEFAULT_TABLE table=default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.303259 1560 raft_consensus.cc:2466] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:09.303360 1637 leader_election.cc:304] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:09.303483 1537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a69c50e5f14364825e048fd19d4409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.303547 2093 raft_consensus.cc:2802] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:09.303598 2093 raft_consensus.cc:491] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:09.303633 2093 raft_consensus.cc:3058] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.303584 1399 tablet_service.cc:1468] Processing CreateTablet for tablet c9a69c50e5f14364825e048fd19d4409 (DEFAULT_TABLE table=default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.303853 1399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a69c50e5f14364825e048fd19d4409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.304320 1675 tablet_service.cc:1468] Processing CreateTablet for tablet c9a69c50e5f14364825e048fd19d4409 (DEFAULT_TABLE table=default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.304543 1675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a69c50e5f14364825e048fd19d4409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.304797 2093 raft_consensus.cc:513] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.305198 2093 leader_election.cc:290] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:09.305694 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a388d1449bf455dbde985974c23e43f" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:09.305820 1424 raft_consensus.cc:3058] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.306535 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a388d1449bf455dbde985974c23e43f" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:09.306586 2056 tablet_bootstrap.cc:492] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:09.306653 1560 raft_consensus.cc:3058] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.306790 1424 raft_consensus.cc:2466] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:09.307149 1637 leader_election.cc:304] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:09.307309 2093 raft_consensus.cc:2802] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:09.307374 2093 raft_consensus.cc:695] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
I20250625 20:01:09.307490 2093 consensus_queue.cc:237] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.307580 2056 tablet_bootstrap.cc:654] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.307561 1560 raft_consensus.cc:2466] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:09.308245 2061 tablet_bootstrap.cc:492] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:09.309186 2061 tablet_bootstrap.cc:654] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.310644 2061 tablet_bootstrap.cc:492] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:09.310724 2061 ts_tablet_manager.cc:1397] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 20:01:09.311198 2061 raft_consensus.cc:357] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:09.311290 2061 raft_consensus.cc:383] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.311321 2061 raft_consensus.cc:738] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.311424 2061 consensus_queue.cc:260] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:09.311678 2061 ts_tablet_manager.cc:1428] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:09.315380 2058 tablet_bootstrap.cc:492] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:09.316442 2058 tablet_bootstrap.cc:654] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.316864 1885 catalog_manager.cc:5582] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1). New cstate: current_term: 1 leader_uuid: "27aa5b53b16747f9a72485aaf2417b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } health_report { overall_health: UNKNOWN } } }
I20250625 20:01:09.318244 2056 tablet_bootstrap.cc:492] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:09.318343 2056 ts_tablet_manager.cc:1397] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20250625 20:01:09.318854 2056 raft_consensus.cc:357] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:09.318953 2056 raft_consensus.cc:383] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.318984 2056 raft_consensus.cc:738] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.319098 2056 consensus_queue.cc:260] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:09.319361 2056 ts_tablet_manager.cc:1428] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:09.322969 1885 catalog_manager.cc:5582] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3). New cstate: current_term: 1 leader_uuid: "0ed894f2ff26414190d33cd134c36bc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } health_report { overall_health: UNKNOWN } } }
I20250625 20:01:09.326666 1881 catalog_manager.cc:2149] created HMS catalog entry for table [id=f57ce585c27c4ac3b416bc59a660690e]
I20250625 20:01:09.327780 2058 tablet_bootstrap.cc:492] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:09.327888 2058 ts_tablet_manager.cc:1397] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250625 20:01:09.328323 2058 raft_consensus.cc:357] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:09.328424 2058 raft_consensus.cc:383] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.328454 2058 raft_consensus.cc:738] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.328575 2058 consensus_queue.cc:260] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:09.328853 2058 ts_tablet_manager.cc:1428] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 20:01:09.333251 1885 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49704:
table { table_name: "default.Table_1d0eb88cbdfd4b16bb66bfb3d502672b" } new_table_name: "default.Table_dd960e4b8a9c42309150252433773c76" modify_external_catalogs: true
I20250625 20:01:09.342535 1675 tablet_service.cc:1468] Processing CreateTablet for tablet c368f57ac08446e5a34b67f11ac51dc8 (DEFAULT_TABLE table=default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.342742 1399 tablet_service.cc:1468] Processing CreateTablet for tablet c368f57ac08446e5a34b67f11ac51dc8 (DEFAULT_TABLE table=default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.342898 1675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c368f57ac08446e5a34b67f11ac51dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.342983 1399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c368f57ac08446e5a34b67f11ac51dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.343041 1537 tablet_service.cc:1468] Processing CreateTablet for tablet c368f57ac08446e5a34b67f11ac51dc8 (DEFAULT_TABLE table=default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.343317 1537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c368f57ac08446e5a34b67f11ac51dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.345139 2056 tablet_bootstrap.cc:492] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:09.346115 1881 master_service.cc:948] ReplaceTablet: received request to replace tablet 1c5e0c439cdb4e67a155bd125982f971 from {username='slave'} at 127.0.0.1:49704
I20250625 20:01:09.346166 2056 tablet_bootstrap.cc:654] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.347260 2061 tablet_bootstrap.cc:492] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:09.347733 2058 tablet_bootstrap.cc:492] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:09.347996 2056 tablet_bootstrap.cc:492] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:09.348070 2056 ts_tablet_manager.cc:1397] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 20:01:09.348284 2061 tablet_bootstrap.cc:654] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.348539 2056 raft_consensus.cc:357] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.348640 2056 raft_consensus.cc:383] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.348670 2056 raft_consensus.cc:738] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.348804 2058 tablet_bootstrap.cc:654] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.348788 2056 consensus_queue.cc:260] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.349659 2056 ts_tablet_manager.cc:1428] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:09.352833 1881 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Sending DeleteTablet for 3 replicas of tablet 1c5e0c439cdb4e67a155bd125982f971
I20250625 20:01:09.353101 2061 tablet_bootstrap.cc:492] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:09.353171 1881 catalog_manager.cc:6652] ReplaceTablet: tablet 1c5e0c439cdb4e67a155bd125982f971 deleted and replaced by tablet d8dd4c0e80974a67b63293d83760fd8b
I20250625 20:01:09.353199 2061 ts_tablet_manager.cc:1397] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20250625 20:01:09.353305 1399 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c5e0c439cdb4e67a155bd125982f971 with delete_type TABLET_DATA_DELETED (replaced by tablet d8dd4c0e80974a67b63293d83760fd8b) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:09.353467 2055 tablet_replica.cc:331] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:09.353587 2055 raft_consensus.cc:2241] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:09.353682 2055 raft_consensus.cc:2270] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:09.353981 1881 master_service.cc:948] ReplaceTablet: received request to replace tablet 44500be7b904460fa32515082b47a6f0 from {username='slave'} at 127.0.0.1:49704
I20250625 20:01:09.354226 2055 ts_tablet_manager.cc:1905] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:09.355165 2061 raft_consensus.cc:357] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.355280 2061 raft_consensus.cc:383] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.355312 2061 raft_consensus.cc:738] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.355460 2061 consensus_queue.cc:260] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.355572 1675 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c5e0c439cdb4e67a155bd125982f971 with delete_type TABLET_DATA_DELETED (replaced by tablet d8dd4c0e80974a67b63293d83760fd8b) from {username='slave'} at 127.0.0.1:43128
I20250625 20:01:09.355773 2060 tablet_replica.cc:331] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:09.355899 2060 raft_consensus.cc:2241] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:09.355999 2060 raft_consensus.cc:2270] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:09.355772 2061 ts_tablet_manager.cc:1428] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 20:01:09.356562 2060 ts_tablet_manager.cc:1905] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:09.357554 2055 ts_tablet_manager.cc:1918] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:09.358378 2055 log.cc:1199] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/1c5e0c439cdb4e67a155bd125982f971
I20250625 20:01:09.358610 2060 ts_tablet_manager.cc:1918] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:09.358623 2055 ts_tablet_manager.cc:1939] T 1c5e0c439cdb4e67a155bd125982f971 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:09.358668 2060 log.cc:1199] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/1c5e0c439cdb4e67a155bd125982f971
I20250625 20:01:09.358775 1850 raft_consensus.cc:491] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:09.358847 1850 raft_consensus.cc:513] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.358896 2060 ts_tablet_manager.cc:1939] T 1c5e0c439cdb4e67a155bd125982f971 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:09.359228 1850 leader_election.cc:290] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:09.359843 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c368f57ac08446e5a34b67f11ac51dc8" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:09.359951 1701 raft_consensus.cc:2466] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:09.359947 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c368f57ac08446e5a34b67f11ac51dc8" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
W20250625 20:01:09.360338 1357 leader_election.cc:343] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Illegal state: must be running to vote when last-logged opid is not known
I20250625 20:01:09.360441 1358 leader_election.cc:304] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters: c73846c0665d4d53a274d053946107df
I20250625 20:01:09.360477 1872 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet 1c5e0c439cdb4e67a155bd125982f971 (table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]) successfully deleted
I20250625 20:01:09.361577 2093 consensus_queue.cc:1035] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.361883 1850 raft_consensus.cc:2802] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:09.361941 1850 raft_consensus.cc:491] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:09.361981 1850 raft_consensus.cc:3058] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.362903 1850 raft_consensus.cc:513] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.363289 1850 leader_election.cc:290] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:09.363653 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c368f57ac08446e5a34b67f11ac51dc8" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
W20250625 20:01:09.364056 1357 leader_election.cc:343] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Illegal state: must be running to vote when last-logged opid is not known
I20250625 20:01:09.364216 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c368f57ac08446e5a34b67f11ac51dc8" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:09.364317 1701 raft_consensus.cc:3058] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.365163 1701 raft_consensus.cc:2466] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:09.365448 1358 leader_election.cc:304] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters: c73846c0665d4d53a274d053946107df
I20250625 20:01:09.365733 1991 raft_consensus.cc:2802] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:09.365806 1991 raft_consensus.cc:695] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:09.365919 1991 consensus_queue.cc:237] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.365972 2058 tablet_bootstrap.cc:492] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:09.366050 2058 ts_tablet_manager.cc:1397] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20250625 20:01:09.366549 2058 raft_consensus.cc:357] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.366613 1535 tablet_service.cc:1468] Processing CreateTablet for tablet d8dd4c0e80974a67b63293d83760fd8b (DEFAULT_TABLE table=default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.366880 1535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8dd4c0e80974a67b63293d83760fd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.368010 1874 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet 1c5e0c439cdb4e67a155bd125982f971 (table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]) successfully deleted
I20250625 20:01:09.366662 2058 raft_consensus.cc:383] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.368170 2058 raft_consensus.cc:738] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.368291 2058 consensus_queue.cc:260] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.368335 1399 tablet_service.cc:1468] Processing CreateTablet for tablet d8dd4c0e80974a67b63293d83760fd8b (DEFAULT_TABLE table=default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.368556 2058 ts_tablet_manager.cc:1428] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:09.368575 1399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8dd4c0e80974a67b63293d83760fd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.369692 2058 tablet_bootstrap.cc:492] T d8dd4c0e80974a67b63293d83760fd8b P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:09.370718 2058 tablet_bootstrap.cc:654] T d8dd4c0e80974a67b63293d83760fd8b P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.371464 1537 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c5e0c439cdb4e67a155bd125982f971 with delete_type TABLET_DATA_DELETED (replaced by tablet d8dd4c0e80974a67b63293d83760fd8b) from {username='slave'} at 127.0.0.1:57380
I20250625 20:01:09.371526 1675 tablet_service.cc:1468] Processing CreateTablet for tablet d8dd4c0e80974a67b63293d83760fd8b (DEFAULT_TABLE table=default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.371704 2057 tablet_replica.cc:331] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:09.371830 1675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8dd4c0e80974a67b63293d83760fd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.371843 2057 raft_consensus.cc:2241] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:09.372022 2057 raft_consensus.cc:2270] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:09.372522 2057 ts_tablet_manager.cc:1905] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:09.374361 1881 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Sending DeleteTablet for 3 replicas of tablet 44500be7b904460fa32515082b47a6f0
I20250625 20:01:09.374547 2057 ts_tablet_manager.cc:1918] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:09.374604 2057 log.cc:1199] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/1c5e0c439cdb4e67a155bd125982f971
I20250625 20:01:09.374819 2057 ts_tablet_manager.cc:1939] T 1c5e0c439cdb4e67a155bd125982f971 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:09.374831 2056 tablet_bootstrap.cc:492] T d8dd4c0e80974a67b63293d83760fd8b P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:09.375207 1881 catalog_manager.cc:6652] ReplaceTablet: tablet 44500be7b904460fa32515082b47a6f0 deleted and replaced by tablet 4ec346c593cc448fbeb6d257ecc8dc52
I20250625 20:01:09.375581 1674 tablet_service.cc:1515] Processing DeleteTablet for tablet 44500be7b904460fa32515082b47a6f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 4ec346c593cc448fbeb6d257ecc8dc52) from {username='slave'} at 127.0.0.1:43128
I20250625 20:01:09.375602 1871 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet 1c5e0c439cdb4e67a155bd125982f971 (table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]) successfully deleted
I20250625 20:01:09.375950 1537 tablet_service.cc:1515] Processing DeleteTablet for tablet 44500be7b904460fa32515082b47a6f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 4ec346c593cc448fbeb6d257ecc8dc52) from {username='slave'} at 127.0.0.1:57380
I20250625 20:01:09.376056 2056 tablet_bootstrap.cc:654] T d8dd4c0e80974a67b63293d83760fd8b P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.377723 1399 tablet_service.cc:1515] Processing DeleteTablet for tablet 44500be7b904460fa32515082b47a6f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 4ec346c593cc448fbeb6d257ecc8dc52) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:09.377910 2060 tablet_replica.cc:331] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:09.378043 2060 raft_consensus.cc:2241] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:09.378208 2060 raft_consensus.cc:2270] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:09.379211 2060 ts_tablet_manager.cc:1905] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:09.379329 2058 tablet_bootstrap.cc:492] T d8dd4c0e80974a67b63293d83760fd8b P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:09.379426 2058 ts_tablet_manager.cc:1397] T d8dd4c0e80974a67b63293d83760fd8b P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250625 20:01:09.379405 1829 consensus_queue.cc:1035] T 78631bcca66d4b16b344e57f8a1cc291 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.379931 2058 raft_consensus.cc:357] T d8dd4c0e80974a67b63293d83760fd8b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.380034 2058 raft_consensus.cc:383] T d8dd4c0e80974a67b63293d83760fd8b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.380065 2058 raft_consensus.cc:738] T d8dd4c0e80974a67b63293d83760fd8b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.381428 2060 ts_tablet_manager.cc:1918] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:09.381490 2060 log.cc:1199] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/44500be7b904460fa32515082b47a6f0
I20250625 20:01:09.381709 2060 ts_tablet_manager.cc:1939] T 44500be7b904460fa32515082b47a6f0 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:09.382489 1872 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet 44500be7b904460fa32515082b47a6f0 (table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]) successfully deleted
I20250625 20:01:09.383205 2057 tablet_replica.cc:331] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:09.383342 2057 raft_consensus.cc:2241] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:09.383447 2057 raft_consensus.cc:2270] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:09.383662 2055 tablet_replica.cc:331] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:09.383818 2057 ts_tablet_manager.cc:1905] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:09.383821 2055 raft_consensus.cc:2241] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:09.383936 2055 raft_consensus.cc:2270] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:09.384258 2055 ts_tablet_manager.cc:1905] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:09.385828 2057 ts_tablet_manager.cc:1918] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:09.385898 2057 log.cc:1199] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/44500be7b904460fa32515082b47a6f0
I20250625 20:01:09.386282 1886 catalog_manager.cc:2149] created HMS catalog entry for table [id=71733c4c432244469f60c9ffeb917cc7]
I20250625 20:01:09.386543 2055 ts_tablet_manager.cc:1918] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:09.386611 2055 log.cc:1199] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/44500be7b904460fa32515082b47a6f0
I20250625 20:01:09.386842 2055 ts_tablet_manager.cc:1939] T 44500be7b904460fa32515082b47a6f0 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:09.387442 2061 tablet_bootstrap.cc:492] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:09.387526 1874 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet 44500be7b904460fa32515082b47a6f0 (table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]) successfully deleted
I20250625 20:01:09.389024 2061 tablet_bootstrap.cc:654] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.389175 1537 tablet_service.cc:1468] Processing CreateTablet for tablet 4ec346c593cc448fbeb6d257ecc8dc52 (DEFAULT_TABLE table=default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.389315 1675 tablet_service.cc:1468] Processing CreateTablet for tablet 4ec346c593cc448fbeb6d257ecc8dc52 (DEFAULT_TABLE table=default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.389410 1537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ec346c593cc448fbeb6d257ecc8dc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.389528 1675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ec346c593cc448fbeb6d257ecc8dc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.389712 1399 tablet_service.cc:1468] Processing CreateTablet for tablet 4ec346c593cc448fbeb6d257ecc8dc52 (DEFAULT_TABLE table=default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.389926 1399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ec346c593cc448fbeb6d257ecc8dc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.391541 2058 consensus_queue.cc:260] T d8dd4c0e80974a67b63293d83760fd8b P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.391896 2058 ts_tablet_manager.cc:1428] T d8dd4c0e80974a67b63293d83760fd8b P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.012s user 0.000s sys 0.001s
I20250625 20:01:09.392071 2058 tablet_bootstrap.cc:492] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:09.392315 2061 tablet_bootstrap.cc:492] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:09.392382 2061 ts_tablet_manager.cc:1397] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 20:01:09.392866 2061 raft_consensus.cc:357] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.392959 2061 raft_consensus.cc:383] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.392988 2061 raft_consensus.cc:738] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.393105 2061 consensus_queue.cc:260] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.393366 2061 ts_tablet_manager.cc:1428] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:09.393549 2061 tablet_bootstrap.cc:492] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:09.394444 2061 tablet_bootstrap.cc:654] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.395350 2057 ts_tablet_manager.cc:1939] T 44500be7b904460fa32515082b47a6f0 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:09.395941 1871 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet 44500be7b904460fa32515082b47a6f0 (table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]) successfully deleted
I20250625 20:01:09.396176 2058 tablet_bootstrap.cc:654] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.396332 1881 catalog_manager.cc:5582] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1). New cstate: current_term: 1 leader_uuid: "27aa5b53b16747f9a72485aaf2417b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } health_report { overall_health: UNKNOWN } } }
I20250625 20:01:09.400276 2061 tablet_bootstrap.cc:492] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:09.400368 2061 ts_tablet_manager.cc:1397] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250625 20:01:09.400853 2061 raft_consensus.cc:357] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.400946 2061 raft_consensus.cc:383] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.400974 2061 raft_consensus.cc:738] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.401087 2061 consensus_queue.cc:260] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.401361 2061 ts_tablet_manager.cc:1428] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:09.403080 2058 tablet_bootstrap.cc:492] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:09.403172 2058 ts_tablet_manager.cc:1397] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250625 20:01:09.403625 2058 raft_consensus.cc:357] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.403765 2058 raft_consensus.cc:383] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.403743 1886 master_service.cc:948] ReplaceTablet: received request to replace tablet d5d6a94383be4a1dba614c0db1b88003 from {username='slave'} at 127.0.0.1:49704
I20250625 20:01:09.403805 2058 raft_consensus.cc:738] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.403945 2058 consensus_queue.cc:260] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.404243 2058 ts_tablet_manager.cc:1428] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:09.405288 2056 tablet_bootstrap.cc:492] T d8dd4c0e80974a67b63293d83760fd8b P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:09.405381 2056 ts_tablet_manager.cc:1397] T d8dd4c0e80974a67b63293d83760fd8b P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20250625 20:01:09.406476 2056 raft_consensus.cc:357] T d8dd4c0e80974a67b63293d83760fd8b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.406584 2056 raft_consensus.cc:383] T d8dd4c0e80974a67b63293d83760fd8b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.406638 2056 raft_consensus.cc:738] T d8dd4c0e80974a67b63293d83760fd8b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.406767 2056 consensus_queue.cc:260] T d8dd4c0e80974a67b63293d83760fd8b P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.407030 2056 ts_tablet_manager.cc:1428] T d8dd4c0e80974a67b63293d83760fd8b P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 20:01:09.407232 2056 tablet_bootstrap.cc:492] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:09.410382 2056 tablet_bootstrap.cc:654] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.412606 2056 tablet_bootstrap.cc:492] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:09.412704 2056 ts_tablet_manager.cc:1397] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250625 20:01:09.413158 2056 raft_consensus.cc:357] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.413264 2056 raft_consensus.cc:383] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.413297 2056 raft_consensus.cc:738] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.413443 2056 consensus_queue.cc:260] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.414757 1886 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Sending DeleteTablet for 3 replicas of tablet d5d6a94383be4a1dba614c0db1b88003
I20250625 20:01:09.415071 1886 catalog_manager.cc:6652] ReplaceTablet: tablet d5d6a94383be4a1dba614c0db1b88003 deleted and replaced by tablet 08697eef3f7f4c45ba3aacf36628e652
I20250625 20:01:09.416790 1535 tablet_service.cc:1468] Processing CreateTablet for tablet ba8f9c5ea37542f3b3f99a0c1d64b90d (DEFAULT_TABLE table=default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.417091 1535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba8f9c5ea37542f3b3f99a0c1d64b90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.419310 2058 tablet_bootstrap.cc:492] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:09.420258 1935 raft_consensus.cc:491] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:09.420367 2058 tablet_bootstrap.cc:654] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.420336 1935 raft_consensus.cc:513] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.420756 1935 leader_election.cc:290] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:09.421216 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdb2a543898642d3abc3afa850178120" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:09.421355 1701 raft_consensus.cc:2466] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:09.421927 1675 tablet_service.cc:1515] Processing DeleteTablet for tablet d5d6a94383be4a1dba614c0db1b88003 with delete_type TABLET_DATA_DELETED (replaced by tablet 08697eef3f7f4c45ba3aacf36628e652) from {username='slave'} at 127.0.0.1:43128
I20250625 20:01:09.422109 1674 tablet_service.cc:1468] Processing CreateTablet for tablet ba8f9c5ea37542f3b3f99a0c1d64b90d (DEFAULT_TABLE table=default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.422135 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdb2a543898642d3abc3afa850178120" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:09.422256 1424 raft_consensus.cc:2466] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:09.422353 1674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba8f9c5ea37542f3b3f99a0c1d64b90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.422457 2060 tablet_replica.cc:331] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:09.422587 2060 raft_consensus.cc:2241] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:09.422719 2060 raft_consensus.cc:2270] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:09.423118 1495 leader_election.cc:304] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:09.424088 2060 ts_tablet_manager.cc:1905] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:09.424872 2061 tablet_bootstrap.cc:492] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:09.425992 2061 tablet_bootstrap.cc:654] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.426119 2060 ts_tablet_manager.cc:1918] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 20:01:09.426182 2060 log.cc:1199] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/d5d6a94383be4a1dba614c0db1b88003
I20250625 20:01:09.426414 2060 ts_tablet_manager.cc:1939] T d5d6a94383be4a1dba614c0db1b88003 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:09.427336 1872 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet d5d6a94383be4a1dba614c0db1b88003 (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]) successfully deleted
I20250625 20:01:09.427497 1537 tablet_service.cc:1515] Processing DeleteTablet for tablet d5d6a94383be4a1dba614c0db1b88003 with delete_type TABLET_DATA_DELETED (replaced by tablet 08697eef3f7f4c45ba3aacf36628e652) from {username='slave'} at 127.0.0.1:57380
I20250625 20:01:09.427667 2057 tablet_replica.cc:331] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:09.427836 2057 raft_consensus.cc:2241] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:09.427899 1935 raft_consensus.cc:2802] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:09.427954 1935 raft_consensus.cc:491] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:09.427974 2057 raft_consensus.cc:2270] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:09.428927 2061 tablet_bootstrap.cc:492] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:09.429003 2061 ts_tablet_manager.cc:1397] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250625 20:01:09.429443 2061 raft_consensus.cc:357] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.429538 2061 raft_consensus.cc:383] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.429569 2061 raft_consensus.cc:738] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.429682 2061 consensus_queue.cc:260] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.429977 2061 ts_tablet_manager.cc:1428] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:09.431030 1881 master_service.cc:948] ReplaceTablet: received request to replace tablet 6d488e29942c46cb8c67058068ce2b52 from {username='slave'} at 127.0.0.1:49704
I20250625 20:01:09.431406 1399 tablet_service.cc:1515] Processing DeleteTablet for tablet d5d6a94383be4a1dba614c0db1b88003 with delete_type TABLET_DATA_DELETED (replaced by tablet 08697eef3f7f4c45ba3aacf36628e652) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:09.431736 1400 tablet_service.cc:1468] Processing CreateTablet for tablet ba8f9c5ea37542f3b3f99a0c1d64b90d (DEFAULT_TABLE table=default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.432096 1400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba8f9c5ea37542f3b3f99a0c1d64b90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.433947 1674 tablet_service.cc:1468] Processing CreateTablet for tablet 08697eef3f7f4c45ba3aacf36628e652 (DEFAULT_TABLE table=default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.434185 1674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08697eef3f7f4c45ba3aacf36628e652. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.431826 2055 tablet_replica.cc:331] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:09.434577 2055 raft_consensus.cc:2241] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:09.434820 2055 raft_consensus.cc:2270] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:09.435003 1886 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49704:
table { table_name: "default.Table_063972c2ee4b4e468cf982177f09f91a" } new_table_name: "default.Table_980a5f7891e24ccc87c72a858cc5076f" modify_external_catalogs: true
I20250625 20:01:09.435703 1537 tablet_service.cc:1468] Processing CreateTablet for tablet 08697eef3f7f4c45ba3aacf36628e652 (DEFAULT_TABLE table=default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.436000 1537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08697eef3f7f4c45ba3aacf36628e652. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.436561 2057 ts_tablet_manager.cc:1905] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:09.438422 1881 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Sending DeleteTablet for 3 replicas of tablet 6d488e29942c46cb8c67058068ce2b52
I20250625 20:01:09.438571 2057 ts_tablet_manager.cc:1918] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 20:01:09.438629 2057 log.cc:1199] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/d5d6a94383be4a1dba614c0db1b88003
I20250625 20:01:09.438715 1881 catalog_manager.cc:6652] ReplaceTablet: tablet 6d488e29942c46cb8c67058068ce2b52 deleted and replaced by tablet 06ef5241d21449169f228cfa37ff91b4
I20250625 20:01:09.438907 2057 ts_tablet_manager.cc:1939] T d5d6a94383be4a1dba614c0db1b88003 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:09.439568 1881 master_service.cc:948] ReplaceTablet: received request to replace tablet b36de85648bb4236a9320412cdb1cfdf from {username='slave'} at 127.0.0.1:49704
I20250625 20:01:09.439814 816 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu with pid 1200
I20250625 20:01:09.439867 2055 ts_tablet_manager.cc:1905] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:09.440590 2056 ts_tablet_manager.cc:1428] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.028s user 0.001s sys 0.000s
I20250625 20:01:09.441862 2055 ts_tablet_manager.cc:1918] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 20:01:09.441926 2055 log.cc:1199] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/d5d6a94383be4a1dba614c0db1b88003
I20250625 20:01:09.442144 2055 ts_tablet_manager.cc:1939] T d5d6a94383be4a1dba614c0db1b88003 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:09.442258 1537 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d488e29942c46cb8c67058068ce2b52 with delete_type TABLET_DATA_DELETED (replaced by tablet 06ef5241d21449169f228cfa37ff91b4) from {username='slave'} at 127.0.0.1:57380
I20250625 20:01:09.442757 1881 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49704:
name: "default.Table_3629c083428e4ae0a977ffb16af83bc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:09.442803 1675 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d488e29942c46cb8c67058068ce2b52 with delete_type TABLET_DATA_DELETED (replaced by tablet 06ef5241d21449169f228cfa37ff91b4) from {username='slave'} at 127.0.0.1:43128
I20250625 20:01:09.442970 2060 tablet_replica.cc:331] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:09.443085 2060 raft_consensus.cc:2241] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:09.443214 2060 raft_consensus.cc:2270] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:09.443580 2060 ts_tablet_manager.cc:1905] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:09.444182 1397 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d488e29942c46cb8c67058068ce2b52 with delete_type TABLET_DATA_DELETED (replaced by tablet 06ef5241d21449169f228cfa37ff91b4) from {username='slave'} at 127.0.0.1:46136
I20250625 20:01:09.444416 1399 tablet_service.cc:1468] Processing CreateTablet for tablet 08697eef3f7f4c45ba3aacf36628e652 (DEFAULT_TABLE table=default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.427990 1935 raft_consensus.cc:3058] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.445413 1935 raft_consensus.cc:513] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.445605 2060 ts_tablet_manager.cc:1918] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 20:01:09.445669 2060 log.cc:1199] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/6d488e29942c46cb8c67058068ce2b52
I20250625 20:01:09.445734 1935 leader_election.cc:290] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:09.445899 2060 ts_tablet_manager.cc:1939] T 6d488e29942c46cb8c67058068ce2b52 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:09.446451 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdb2a543898642d3abc3afa850178120" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:09.446559 1701 raft_consensus.cc:3058] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.447363 1701 raft_consensus.cc:2466] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:09.447840 1495 leader_election.cc:304] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:09.448230 1872 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet 6d488e29942c46cb8c67058068ce2b52 (table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996]) successfully deleted
I20250625 20:01:09.448386 1935 raft_consensus.cc:2802] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:09.448468 1935 raft_consensus.cc:695] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Becoming Leader. State: Replica: c73846c0665d4d53a274d053946107df, State: Running, Role: LEADER
I20250625 20:01:09.448603 1935 consensus_queue.cc:237] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.450717 1397 tablet_service.cc:1468] Processing CreateTablet for tablet 06ef5241d21449169f228cfa37ff91b4 (DEFAULT_TABLE table=default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.450951 1537 tablet_service.cc:1468] Processing CreateTablet for tablet 06ef5241d21449169f228cfa37ff91b4 (DEFAULT_TABLE table=default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.451191 1537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06ef5241d21449169f228cfa37ff91b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.451329 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdb2a543898642d3abc3afa850178120" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:09.451434 1424 raft_consensus.cc:3058] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.452350 1424 raft_consensus.cc:2466] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:09.452623 1399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08697eef3f7f4c45ba3aacf36628e652. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.454977 1674 tablet_service.cc:1468] Processing CreateTablet for tablet 06ef5241d21449169f228cfa37ff91b4 (DEFAULT_TABLE table=default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:09.455072 2056 tablet_bootstrap.cc:492] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:09.455329 1674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06ef5241d21449169f228cfa37ff91b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.455958 1397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06ef5241d21449169f228cfa37ff91b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:09.456377 2056 tablet_bootstrap.cc:654] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.457909 2056 tablet_bootstrap.cc:492] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:09.457999 2056 ts_tablet_manager.cc:1397] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 20:01:09.458487 2061 tablet_bootstrap.cc:492] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:09.458472 2056 raft_consensus.cc:357] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.458567 2056 raft_consensus.cc:383] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.458597 2056 raft_consensus.cc:738] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.458732 2056 consensus_queue.cc:260] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.459067 2056 ts_tablet_manager.cc:1428] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:09.459223 2056 tablet_bootstrap.cc:492] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:09.459478 2061 tablet_bootstrap.cc:654] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.459967 1871 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet d5d6a94383be4a1dba614c0db1b88003 (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]) successfully deleted
I20250625 20:01:09.460130 2056 tablet_bootstrap.cc:654] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.460494 2055 tablet_replica.cc:331] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:09.460619 2055 raft_consensus.cc:2241] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:09.460736 2055 raft_consensus.cc:2270] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:09.461078 2055 ts_tablet_manager.cc:1905] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:09.462936 2055 ts_tablet_manager.cc:1918] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 20:01:09.463013 2055 log.cc:1199] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/6d488e29942c46cb8c67058068ce2b52
I20250625 20:01:09.463265 2055 ts_tablet_manager.cc:1939] T 6d488e29942c46cb8c67058068ce2b52 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:09.463945 1874 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet d5d6a94383be4a1dba614c0db1b88003 (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]) successfully deleted
I20250625 20:01:09.464123 1874 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet 6d488e29942c46cb8c67058068ce2b52 (table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996]) successfully deleted
I20250625 20:01:09.464339 2057 tablet_replica.cc:331] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:09.464475 2057 raft_consensus.cc:2241] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:09.464679 2057 raft_consensus.cc:2270] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:09.466986 2057 ts_tablet_manager.cc:1905] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:09.467971 2056 tablet_bootstrap.cc:492] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:09.468083 2056 ts_tablet_manager.cc:1397] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250625 20:01:09.468595 2056 raft_consensus.cc:357] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.468781 2056 raft_consensus.cc:383] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.468824 2056 raft_consensus.cc:738] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.468958 2056 consensus_queue.cc:260] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.469266 2056 ts_tablet_manager.cc:1428] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:09.469462 2056 tablet_bootstrap.cc:492] T 06ef5241d21449169f228cfa37ff91b4 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:09.470464 2056 tablet_bootstrap.cc:654] T 06ef5241d21449169f228cfa37ff91b4 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.470902 2061 tablet_bootstrap.cc:492] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:09.471051 2061 ts_tablet_manager.cc:1397] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250625 20:01:09.471565 2061 raft_consensus.cc:357] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.471737 2061 raft_consensus.cc:383] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.471827 2061 raft_consensus.cc:738] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.472039 2061 consensus_queue.cc:260] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.472676 2056 tablet_bootstrap.cc:492] T 06ef5241d21449169f228cfa37ff91b4 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:09.472750 2056 ts_tablet_manager.cc:1397] T 06ef5241d21449169f228cfa37ff91b4 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 20:01:09.473011 2061 ts_tablet_manager.cc:1428] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:09.473269 2061 tablet_bootstrap.cc:492] T 06ef5241d21449169f228cfa37ff91b4 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:09.473280 2056 raft_consensus.cc:357] T 06ef5241d21449169f228cfa37ff91b4 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.473390 2056 raft_consensus.cc:383] T 06ef5241d21449169f228cfa37ff91b4 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.473426 2056 raft_consensus.cc:738] T 06ef5241d21449169f228cfa37ff91b4 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.473531 2056 consensus_queue.cc:260] T 06ef5241d21449169f228cfa37ff91b4 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.473770 2056 ts_tablet_manager.cc:1428] T 06ef5241d21449169f228cfa37ff91b4 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 20:01:09.475486 2057 ts_tablet_manager.cc:1918] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 20:01:09.475564 2057 log.cc:1199] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/6d488e29942c46cb8c67058068ce2b52
I20250625 20:01:09.476356 2057 ts_tablet_manager.cc:1939] T 6d488e29942c46cb8c67058068ce2b52 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:09.477088 1871 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet 6d488e29942c46cb8c67058068ce2b52 (table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996]) successfully deleted
I20250625 20:01:09.477706 2061 tablet_bootstrap.cc:654] T 06ef5241d21449169f228cfa37ff91b4 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.478665 2061 tablet_bootstrap.cc:492] T 06ef5241d21449169f228cfa37ff91b4 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:09.478746 2061 ts_tablet_manager.cc:1397] T 06ef5241d21449169f228cfa37ff91b4 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
W20250625 20:01:09.479059 1637 connection.cc:537] client connection to 127.0.204.60:46191 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:09.479122 1360 connection.cc:537] client connection to 127.0.204.60:46191 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:09.479135 1874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.204.60:46191 (error 108)
W20250625 20:01:09.479218 1497 connection.cc:537] client connection to 127.0.204.60:46191 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 20:01:09.479629 2061 raft_consensus.cc:357] T 06ef5241d21449169f228cfa37ff91b4 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.479800 2061 raft_consensus.cc:383] T 06ef5241d21449169f228cfa37ff91b4 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.479846 2061 raft_consensus.cc:738] T 06ef5241d21449169f228cfa37ff91b4 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.480001 2061 consensus_queue.cc:260] T 06ef5241d21449169f228cfa37ff91b4 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.480515 2058 tablet_bootstrap.cc:492] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:09.480594 2058 ts_tablet_manager.cc:1397] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.000s
I20250625 20:01:09.481074 2058 raft_consensus.cc:357] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.481184 2058 raft_consensus.cc:383] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.481215 2058 raft_consensus.cc:738] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.481338 2058 consensus_queue.cc:260] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.481622 2058 ts_tablet_manager.cc:1428] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:09.481832 2058 tablet_bootstrap.cc:492] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:09.482504 2061 ts_tablet_manager.cc:1428] T 06ef5241d21449169f228cfa37ff91b4 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 20:01:09.482791 2058 tablet_bootstrap.cc:654] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.483884 2058 tablet_bootstrap.cc:492] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:09.484017 2058 ts_tablet_manager.cc:1397] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
W20250625 20:01:09.484866 2129 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 20:01:09.480259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.204.60:46191 (local address 127.0.0.1:47294)
0625 20:01:09.484212 (+ 3953us) negotiation.cc:107] Waiting for socket to connect
0625 20:01:09.484289 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":3855,"thread_start_us":51,"threads_started":1}
I20250625 20:01:09.485491 2058 raft_consensus.cc:357] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.485684 2058 raft_consensus.cc:383] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 20:01:09.486379 2128 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 20:01:09.479678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.204.60:46191 (local address 127.0.204.1:58537)
0625 20:01:09.485729 (+ 6051us) negotiation.cc:107] Waiting for socket to connect
0625 20:01:09.485790 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":5956,"thread_start_us":101,"threads_started":1}
I20250625 20:01:09.486503 2058 raft_consensus.cc:738] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.486706 2058 consensus_queue.cc:260] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.487051 2058 ts_tablet_manager.cc:1428] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250625 20:01:09.487293 2058 tablet_bootstrap.cc:492] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:09.489540 2058 tablet_bootstrap.cc:654] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:09.485723 1888 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]
W20250625 20:01:09.490072 1874 consensus_peers.cc:487] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 -> Peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Couldn't send request to peer 1fcdee0c0aac4013a5e3c77304b474d9. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20250625 20:01:09.491770 2058 tablet_bootstrap.cc:492] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:09.491865 2058 ts_tablet_manager.cc:1397] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 20:01:09.492339 2058 raft_consensus.cc:357] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.492450 2058 raft_consensus.cc:383] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:09.492487 2058 raft_consensus.cc:738] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:09.492632 2058 consensus_queue.cc:260] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.492914 2058 ts_tablet_manager.cc:1428] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:09.493068 1850 raft_consensus.cc:491] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:09.493160 1850 raft_consensus.cc:513] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:09.493566 1850 leader_election.cc:290] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:09.494068 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9a69c50e5f14364825e048fd19d4409" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:09.494138 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9a69c50e5f14364825e048fd19d4409" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:09.494206 1701 raft_consensus.cc:2466] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:09.494226 1560 raft_consensus.cc:2466] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:09.494482 1357 leader_election.cc:304] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:09.494638 1850 raft_consensus.cc:2802] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:09.494681 1850 raft_consensus.cc:491] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:09.494715 1850 raft_consensus.cc:3058] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.495738 1850 raft_consensus.cc:513] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:09.496088 1850 leader_election.cc:290] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:09.496433 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9a69c50e5f14364825e048fd19d4409" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:09.496434 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9a69c50e5f14364825e048fd19d4409" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:09.496531 1701 raft_consensus.cc:3058] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.496531 1560 raft_consensus.cc:3058] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.497385 1701 raft_consensus.cc:2466] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:09.497385 1560 raft_consensus.cc:2466] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:09.497655 1357 leader_election.cc:304] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:09.497840 1850 raft_consensus.cc:2802] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:09.497913 1850 raft_consensus.cc:695] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:09.498070 1850 consensus_queue.cc:237] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
F20250625 20:01:09.501186 1954 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 6 out of 13 received events: NotificationEvent(eventId=22, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_063972c2ee4b4e468cf982177f09f91a, message={"server":"","servicePrincipal":"","db":"default","table":"table_063972c2ee4b4e468cf982177f09f91a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_063972c2ee4b4e468cf982177f09f91a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_063972c2ee4b4e468cf982177f09f91a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_063972c2ee4b4e468cf982177f09f91a\",\"kudu.table_id\":\"a054c42074394f1db21dfcf2c55d99b5\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_01c13560571f419e9b6f45edee3a2a37, message={"server":"","servicePrincipal":"","db":"default","table":"table_01c13560571f419e9b6f45edee3a2a37","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_01c13560571f419e9b6f45edee3a2a37\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_01c13560571f419e9b6f45edee3a2a37\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_01c13560571f419e9b6f45edee3a2a37\",\"kudu.table_id\":\"aee290f61c8f4d7b93d4cd7192eb9277\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_a31a093a024346aab31bc8d55a00253d, message={"server":"","servicePrincipal":"","db":"default","table":"table_a31a093a024346aab31bc8d55a00253d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a31a093a024346aab31bc8d55a00253d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_a31a093a024346aab31bc8d55a00253d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a31a093a024346aab31bc8d55a00253d\",\"kudu.table_id\":\"81aee385d1404c82aaeebee220cb9e68\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_60554f72b1aa48aa93aaea511a985fe7, message={"server":"","servicePrincipal":"","db":"default","table":"table_60554f72b1aa48aa93aaea511a985fe7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_60554f72b1aa48aa93aaea511a985fe7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881667},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_60554f72b1aa48aa93aaea511a985fe7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_60554f72b1aa48aa93aaea511a985fe7\",\"kudu.table_id\":\"e46f78618ca648e19c3935d0bf874a35\",\"transient_lastDdlTime\":\"1750881667\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f3d1fbe5dcd google::LogMessage::Fail() at ??:0
@ 0x7f3d1fbe9b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f3d1fbe57cc google::LogMessage::Flush() at ??:0
@ 0x7f3d1fbe5a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f3d24b7ec91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f3d24b7dba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f3d24b7d30b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
@ 0x7f3d24b82a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x560dad84d752 std::function<>::operator()() at ??:0
@ 0x7f3d210370a1 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3d218266db start_thread at ??:0
@ 0x7f3d1ed9c71f clone at ??:0
I20250625 20:01:09.603612 1829 raft_consensus.cc:491] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:09.603770 1829 raft_consensus.cc:513] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.604244 1829 leader_election.cc:290] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:09.604707 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "08697eef3f7f4c45ba3aacf36628e652" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:09.604749 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "08697eef3f7f4c45ba3aacf36628e652" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:09.604851 1424 raft_consensus.cc:2466] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:09.604933 1560 raft_consensus.cc:2466] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:09.605247 1634 leader_election.cc:304] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:09.605441 1829 raft_consensus.cc:2802] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:09.605496 1829 raft_consensus.cc:491] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:09.605523 1829 raft_consensus.cc:3058] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.606608 1829 raft_consensus.cc:513] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.606953 1829 leader_election.cc:290] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:09.607276 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "08697eef3f7f4c45ba3aacf36628e652" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:09.607326 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "08697eef3f7f4c45ba3aacf36628e652" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:09.607414 1560 raft_consensus.cc:3058] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.607398 1424 raft_consensus.cc:3058] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.608346 1560 raft_consensus.cc:2466] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:09.608346 1424 raft_consensus.cc:2466] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:09.608618 1637 leader_election.cc:304] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:09.608820 1829 raft_consensus.cc:2802] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:09.608916 1829 raft_consensus.cc:695] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
I20250625 20:01:09.609051 1829 consensus_queue.cc:237] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.611930 1829 raft_consensus.cc:491] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:09.612015 1829 raft_consensus.cc:513] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.612326 1829 leader_election.cc:290] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:09.612896 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba8f9c5ea37542f3b3f99a0c1d64b90d" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:09.612984 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba8f9c5ea37542f3b3f99a0c1d64b90d" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:09.613018 1560 raft_consensus.cc:2466] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:09.613099 1424 raft_consensus.cc:2466] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:09.613365 1637 leader_election.cc:304] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:09.613517 1829 raft_consensus.cc:2802] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:09.613559 1829 raft_consensus.cc:491] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:09.613591 1829 raft_consensus.cc:3058] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.614799 1829 raft_consensus.cc:513] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.615129 1829 leader_election.cc:290] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:09.615460 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba8f9c5ea37542f3b3f99a0c1d64b90d" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:09.615561 1560 raft_consensus.cc:3058] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.616281 1560 raft_consensus.cc:2466] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:09.616480 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba8f9c5ea37542f3b3f99a0c1d64b90d" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:09.616580 1424 raft_consensus.cc:3058] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.617246 1424 raft_consensus.cc:2466] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:09.617520 1637 leader_election.cc:304] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:09.617712 1829 raft_consensus.cc:2802] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:09.617782 1829 raft_consensus.cc:695] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
I20250625 20:01:09.617913 1829 consensus_queue.cc:237] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.654654 2079 raft_consensus.cc:491] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:09.654793 2079 raft_consensus.cc:513] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.655246 2079 leader_election.cc:290] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:09.655655 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c368f57ac08446e5a34b67f11ac51dc8" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:09.655753 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c368f57ac08446e5a34b67f11ac51dc8" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:09.655957 1701 raft_consensus.cc:2391] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c73846c0665d4d53a274d053946107df in current term 1: Already voted for candidate 27aa5b53b16747f9a72485aaf2417b08 in this term.
I20250625 20:01:09.656242 1495 leader_election.cc:304] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c73846c0665d4d53a274d053946107df; no voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08
I20250625 20:01:09.656414 2079 raft_consensus.cc:3058] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.657320 2079 raft_consensus.cc:2747] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:01:09.670116 1850 raft_consensus.cc:491] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:09.670284 1850 raft_consensus.cc:513] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.670838 1850 leader_election.cc:290] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:09.671303 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ec346c593cc448fbeb6d257ecc8dc52" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:09.671478 1560 raft_consensus.cc:2466] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:09.671834 1357 leader_election.cc:304] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:09.671803 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ec346c593cc448fbeb6d257ecc8dc52" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:09.671959 1701 raft_consensus.cc:2466] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:09.672194 1850 raft_consensus.cc:2802] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:09.672266 1850 raft_consensus.cc:491] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:09.672315 1850 raft_consensus.cc:3058] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.673492 1850 raft_consensus.cc:513] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.673949 1850 leader_election.cc:290] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:09.674278 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ec346c593cc448fbeb6d257ecc8dc52" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:09.674417 1701 raft_consensus.cc:3058] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.674379 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ec346c593cc448fbeb6d257ecc8dc52" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:09.674510 1560 raft_consensus.cc:3058] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.675266 1560 raft_consensus.cc:2466] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:09.675426 1701 raft_consensus.cc:2466] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:09.675524 1357 leader_election.cc:304] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:09.675704 1850 raft_consensus.cc:2802] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:09.675874 1850 raft_consensus.cc:695] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:09.676046 1850 consensus_queue.cc:237] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.709414 1829 consensus_queue.cc:1035] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 20:01:09.711377 1754 connection.cc:537] client connection to 127.0.204.61:38005 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:09.711678 1754 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.204.61:38005)
W20250625 20:01:09.711989 1754 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.204.61:38005)
W20250625 20:01:09.712735 1754 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.204.61:38005)
W20250625 20:01:09.712932 1754 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.204.61:38005)
I20250625 20:01:09.713585 1843 consensus_queue.cc:1035] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.751224 2079 raft_consensus.cc:491] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:09.751361 2079 raft_consensus.cc:513] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.751803 2079 leader_election.cc:290] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:09.752214 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06ef5241d21449169f228cfa37ff91b4" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:09.752274 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06ef5241d21449169f228cfa37ff91b4" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:09.752357 1424 raft_consensus.cc:2466] T 06ef5241d21449169f228cfa37ff91b4 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:09.752409 1701 raft_consensus.cc:2466] T 06ef5241d21449169f228cfa37ff91b4 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:09.752640 1497 leader_election.cc:304] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:09.752849 2079 raft_consensus.cc:2802] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:09.752924 2079 raft_consensus.cc:491] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:09.752971 2079 raft_consensus.cc:3058] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.754024 2079 raft_consensus.cc:513] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.754344 2079 leader_election.cc:290] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:09.754721 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06ef5241d21449169f228cfa37ff91b4" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:09.754699 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06ef5241d21449169f228cfa37ff91b4" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:09.754855 1424 raft_consensus.cc:3058] T 06ef5241d21449169f228cfa37ff91b4 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.754855 1701 raft_consensus.cc:3058] T 06ef5241d21449169f228cfa37ff91b4 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.755836 1424 raft_consensus.cc:2466] T 06ef5241d21449169f228cfa37ff91b4 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:09.755861 1701 raft_consensus.cc:2466] T 06ef5241d21449169f228cfa37ff91b4 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:09.756134 1495 leader_election.cc:304] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:09.756366 2079 raft_consensus.cc:2802] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:09.756474 2079 raft_consensus.cc:695] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Becoming Leader. State: Replica: c73846c0665d4d53a274d053946107df, State: Running, Role: LEADER
I20250625 20:01:09.756623 2079 consensus_queue.cc:237] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } }
I20250625 20:01:09.804977 1843 raft_consensus.cc:491] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:09.805116 1843 raft_consensus.cc:513] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.805502 1843 leader_election.cc:290] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:09.805871 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8dd4c0e80974a67b63293d83760fd8b" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:09.805872 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8dd4c0e80974a67b63293d83760fd8b" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:09.806007 1424 raft_consensus.cc:2466] T d8dd4c0e80974a67b63293d83760fd8b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:09.806007 1560 raft_consensus.cc:2466] T d8dd4c0e80974a67b63293d83760fd8b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:09.806289 1634 leader_election.cc:304] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:09.806449 1843 raft_consensus.cc:2802] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:09.806517 1843 raft_consensus.cc:491] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:09.806562 1843 raft_consensus.cc:3058] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.807343 1843 raft_consensus.cc:513] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.807660 1843 leader_election.cc:290] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:09.807983 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8dd4c0e80974a67b63293d83760fd8b" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:09.808094 1560 raft_consensus.cc:3058] T d8dd4c0e80974a67b63293d83760fd8b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.808102 1850 consensus_queue.cc:1035] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.808164 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8dd4c0e80974a67b63293d83760fd8b" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:09.808251 1424 raft_consensus.cc:3058] T d8dd4c0e80974a67b63293d83760fd8b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:09.809026 1424 raft_consensus.cc:2466] T d8dd4c0e80974a67b63293d83760fd8b P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:09.809016 1560 raft_consensus.cc:2466] T d8dd4c0e80974a67b63293d83760fd8b P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:09.809329 1634 leader_election.cc:304] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:09.809482 1843 raft_consensus.cc:2802] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:09.809571 1843 raft_consensus.cc:695] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
I20250625 20:01:09.809726 1843 consensus_queue.cc:237] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } }
I20250625 20:01:09.812938 1850 consensus_queue.cc:1035] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.821542 1850 consensus_queue.cc:1035] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.825865 1991 consensus_queue.cc:1035] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.894728 1991 consensus_queue.cc:1035] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.903591 1935 consensus_queue.cc:1035] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.906909 2079 consensus_queue.cc:1035] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.922148 1850 consensus_queue.cc:1035] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:09.950618 2167 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f87f9e18e5b948a4a3fb3666f106abd0)
I20250625 20:01:09.950742 2167 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:09.951177 2167 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:09.951769 1078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20250625 20:01:09.952566 1078 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:01:09.952677 1075 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:09.952759 1075 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:09.952911 2167 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 20:01:10.098758 1850 consensus_queue.cc:1035] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:10.102372 1991 consensus_queue.cc:1035] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:10.144897 2093 consensus_queue.cc:1035] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:10.148226 1829 consensus_queue.cc:1035] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:10.163394 1829 consensus_queue.cc:1035] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:10.167982 1843 consensus_queue.cc:1035] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:10.185562 1060 master_runner.cc:186] Error getting master registration for 127.0.204.61:38005: Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111),
I20250625 20:01:10.185679 1060 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:10.240178 2079 consensus_queue.cc:1035] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:10.244129 2079 consensus_queue.cc:1035] T 06ef5241d21449169f228cfa37ff91b4 P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:10.317534 1829 consensus_queue.cc:1035] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:10.322268 1843 consensus_queue.cc:1035] T d8dd4c0e80974a67b63293d83760fd8b P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:10.453292 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.60:46191
--webserver_interface=127.0.204.60
--webserver_port=43997
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 20:01:10.559662 2186 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f87f9e18e5b948a4a3fb3666f106abd0)
I20250625 20:01:10.559810 2186 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:10.560288 2186 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:10.561308 1075 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
W20250625 20:01:10.561393 1078 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:10.561444 1078 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0
W20250625 20:01:10.561342 2182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20250625 20:01:10.561575 2186 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250625 20:01:10.561621 2182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:10.561676 2182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:01:10.564899 2182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:01:10.564957 2182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:10.564976 2182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:01:10.564992 2182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:01:10.568740 2182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.60:46191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.204.60
--webserver_port=43997
--never_fsync=true
--heap_profile_path=/tmp/kudu.2182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:10.569765 2182 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:10.570847 2182 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:01:10.577488 2191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:10.577687 2182 server_base.cc:1048] running on GCE node
W20250625 20:01:10.577534 2190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:10.577490 2193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:10.578796 2182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:10.579744 2182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:10.580897 2182 hybrid_clock.cc:648] HybridClock initialized: now 1750881670580870 us; error 46 us; skew 500 ppm
I20250625 20:01:10.586472 2182 webserver.cc:469] Webserver started at http://127.0.204.60:43997/ using document root <none> and password file <none>
I20250625 20:01:10.587076 2182 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:10.587164 2182 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:10.590823 2182 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250625 20:01:10.593171 2199 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:10.594223 2182 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:10.594341 2182 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
uuid: "1fcdee0c0aac4013a5e3c77304b474d9"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
I20250625 20:01:10.594749 2182 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:10.609086 2182 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:10.609710 2182 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:10.609869 2182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:10.616892 2182 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.60:46191
I20250625 20:01:10.616892 2251 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.60:46191 every 8 connection(s)
I20250625 20:01:10.618079 2182 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/info.pb
I20250625 20:01:10.619119 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 2182
I20250625 20:01:10.622762 2252 sys_catalog.cc:263] Verifying existing consensus state
I20250625 20:01:10.624799 2252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Bootstrap starting.
I20250625 20:01:10.634833 2252 log.cc:826] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:10.678404 2252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=84 overwritten=0 applied=83 ignored=0} inserts{seen=57 ignored=0} mutations{seen=67 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250625 20:01:10.678879 2252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Bootstrap complete.
I20250625 20:01:10.681836 2252 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:10.683549 2252 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcdee0c0aac4013a5e3c77304b474d9, State: Initialized, Role: FOLLOWER
I20250625 20:01:10.684136 2252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 83, Last appended: 4.84, Last appended by leader: 84, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:10.684592 2257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:10.684818 2257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:10.685379 2252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:01:10.688099 1470 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:10.690303 2216 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" instance_seqno: 1750881665574847) as {username='slave'} at 127.0.204.1:57661; Asking this server to re-register.
I20250625 20:01:10.692576 1470 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:10.692633 2271 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:10.692703 2271 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: loading cluster ID for follower catalog manager: success
I20250625 20:01:10.693912 2216 ts_manager.cc:194] Registered new tserver with Master: 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:10.695622 2271 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: acquiring CA information for follower catalog manager: success
I20250625 20:01:10.697618 2271 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 20:01:10.706199 2182 master_runner.cc:186] Error getting master registration for 127.0.204.61:38005: Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111),
I20250625 20:01:10.707656 2182 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:10.735807 1607 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:10.737107 2216 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c73846c0665d4d53a274d053946107df" instance_seqno: 1750881665791183) as {username='slave'} at 127.0.204.2:43359; Asking this server to re-register.
I20250625 20:01:10.737596 1607 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:10.738633 2216 ts_manager.cc:194] Registered new tserver with Master: c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:10.746973 1747 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:10.748633 2216 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" instance_seqno: 1750881665979916) as {username='slave'} at 127.0.204.3:47917; Asking this server to re-register.
I20250625 20:01:10.749169 1747 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:10.750227 2216 ts_manager.cc:194] Registered new tserver with Master: 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:11.294418 2276 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:11.294648 2276 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:11.296020 2276 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
I20250625 20:01:11.296468 1101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 84 } } ignore_live_leader: false dest_uuid: "5852fb53503b438c92f294e0c44c916d" is_pre_election: true
I20250625 20:01:11.296641 1101 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fcdee0c0aac4013a5e3c77304b474d9 for term 5 because replica has last-logged OpId of term: 4 index: 86, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 84.
W20250625 20:01:11.296630 2200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
W20250625 20:01:11.297392 2200 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:11.297468 2200 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9; no voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:11.297668 2276 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 20:01:11.573951 2281 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f87f9e18e5b948a4a3fb3666f106abd0)
I20250625 20:01:11.574096 2281 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:11.574463 2281 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:11.575845 1075 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:11.578591 2226 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 5 candidate_status { last_received { term: 4 index: 86 } } ignore_live_leader: false dest_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" is_pre_election: true
I20250625 20:01:11.578860 2226 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5852fb53503b438c92f294e0c44c916d in term 4.
I20250625 20:01:11.579182 1078 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d; no voters: f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:11.579348 2281 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250625 20:01:11.579412 2281 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 4 FOLLOWER]: Starting leader election (detected failure of leader f87f9e18e5b948a4a3fb3666f106abd0)
I20250625 20:01:11.579453 2281 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 4 FOLLOWER]: Advancing to term 5
I20250625 20:01:11.580389 2281 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:11.580731 2281 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 5 election: Requested vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:11.581094 2226 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 5 candidate_status { last_received { term: 4 index: 86 } } ignore_live_leader: false dest_uuid: "1fcdee0c0aac4013a5e3c77304b474d9"
I20250625 20:01:11.581215 2226 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 4 FOLLOWER]: Advancing to term 5
I20250625 20:01:11.583271 2226 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5852fb53503b438c92f294e0c44c916d in term 5.
I20250625 20:01:11.583570 1078 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d; no voters:
I20250625 20:01:11.583746 2281 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 5 FOLLOWER]: Leader election won for term 5
I20250625 20:01:11.583974 2281 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 5 LEADER]: Becoming Leader. State: Replica: 5852fb53503b438c92f294e0c44c916d, State: Running, Role: LEADER
I20250625 20:01:11.584165 2281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 4.86, Last appended by leader: 86, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
W20250625 20:01:11.584287 1075 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:11.584851 2281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5852fb53503b438c92f294e0c44c916d. Latest consensus state: current_term: 5 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:11.584954 2281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:11.585219 2288 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:01:11.586973 2288 catalog_manager.cc:671] Loaded metadata for table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]
I20250625 20:01:11.587486 2288 catalog_manager.cc:671] Loaded metadata for table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]
I20250625 20:01:11.587695 2288 catalog_manager.cc:671] Loaded metadata for table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]
I20250625 20:01:11.587935 2288 catalog_manager.cc:671] Loaded metadata for table default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4]
I20250625 20:01:11.588142 2288 catalog_manager.cc:671] Loaded metadata for table default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]
I20250625 20:01:11.588346 2288 catalog_manager.cc:671] Loaded metadata for table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]
I20250625 20:01:11.588531 2288 catalog_manager.cc:671] Loaded metadata for table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996]
I20250625 20:01:11.588712 2288 catalog_manager.cc:671] Loaded metadata for table default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5]
I20250625 20:01:11.588879 2288 catalog_manager.cc:671] Loaded metadata for table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583]
I20250625 20:01:11.589059 2288 catalog_manager.cc:671] Loaded metadata for table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277]
I20250625 20:01:11.589349 2288 catalog_manager.cc:671] Loaded metadata for table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]
I20250625 20:01:11.589514 2288 catalog_manager.cc:671] Loaded metadata for table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f]
I20250625 20:01:11.589658 2288 catalog_manager.cc:671] Loaded metadata for table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]
I20250625 20:01:11.589800 2288 catalog_manager.cc:671] Loaded metadata for table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]
I20250625 20:01:11.589948 2288 catalog_manager.cc:671] Loaded metadata for table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]
I20250625 20:01:11.590855 2288 tablet_loader.cc:96] loaded metadata for tablet 06ef5241d21449169f228cfa37ff91b4 (table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996])
I20250625 20:01:11.591017 2288 tablet_loader.cc:96] loaded metadata for tablet 07686deb2e184c9fbae06fef5365d0d5 (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0])
I20250625 20:01:11.591176 2288 tablet_loader.cc:96] loaded metadata for tablet 08697eef3f7f4c45ba3aacf36628e652 (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1])
I20250625 20:01:11.591427 2288 tablet_loader.cc:96] loaded metadata for tablet 21677e0e47df421c92f98ed580bbe7a7 (table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8])
I20250625 20:01:11.591576 2288 tablet_loader.cc:96] loaded metadata for tablet 2a388d1449bf455dbde985974c23e43f (table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f])
I20250625 20:01:11.591931 2288 tablet_loader.cc:96] loaded metadata for tablet 4ec346c593cc448fbeb6d257ecc8dc52 (table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35])
I20250625 20:01:11.592168 2288 tablet_loader.cc:96] loaded metadata for tablet 6c8e829c96474e94a8de9e7e32b48350 (table default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4])
I20250625 20:01:11.592478 2288 tablet_loader.cc:96] loaded metadata for tablet 78631bcca66d4b16b344e57f8a1cc291 (table default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5])
I20250625 20:01:11.592994 2288 tablet_loader.cc:96] loaded metadata for tablet b78b6f29890140de958cebed9868b303 (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809])
I20250625 20:01:11.593138 2288 tablet_loader.cc:96] loaded metadata for tablet ba8f9c5ea37542f3b3f99a0c1d64b90d (table default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7])
I20250625 20:01:11.593284 2288 tablet_loader.cc:96] loaded metadata for tablet bb6c755fce6640d2a0b13a449d3adfb9 (table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583])
I20250625 20:01:11.593436 2288 tablet_loader.cc:96] loaded metadata for tablet c368f57ac08446e5a34b67f11ac51dc8 (table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e])
I20250625 20:01:11.593582 2288 tablet_loader.cc:96] loaded metadata for tablet c9a69c50e5f14364825e048fd19d4409 (table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa])
I20250625 20:01:11.593731 2288 tablet_loader.cc:96] loaded metadata for tablet cdb2a543898642d3abc3afa850178120 (table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277])
I20250625 20:01:11.593961 2288 tablet_loader.cc:96] loaded metadata for tablet d8dd4c0e80974a67b63293d83760fd8b (table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36])
I20250625 20:01:11.594249 2288 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:01:11.594929 2288 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:11.594972 2288 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:01:11.595760 2288 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:01:11.596400 2288 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Loaded TSK: 0
I20250625 20:01:11.596643 2288 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:01:11.597178 2288 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 20:01:11.597720 2288 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 21
I20250625 20:01:11.643790 1091 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_a31a093a024346aab31bc8d55a00253d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:11.643898 1089 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_d4ed5ace6c144df6af9ff3155b29c5bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:11.644129 1085 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_3ef4196a2a204be1bb572abdbef3264c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:11.644188 1084 master_service.cc:948] ReplaceTablet: received request to replace tablet 21677e0e47df421c92f98ed580bbe7a7 from {username='slave'} at 127.0.0.1:41858
I20250625 20:01:11.644304 1083 master_service.cc:948] ReplaceTablet: received request to replace tablet 07686deb2e184c9fbae06fef5365d0d5 from {username='slave'} at 127.0.0.1:41858
I20250625 20:01:11.644344 1090 master_service.cc:948] ReplaceTablet: received request to replace tablet b78b6f29890140de958cebed9868b303 from {username='slave'} at 127.0.0.1:41858
I20250625 20:01:11.644590 1086 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_ac867d9f2ac54f09a848a71cd7a01d49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:11.644768 1088 master_service.cc:948] ReplaceTablet: received request to replace tablet 07686deb2e184c9fbae06fef5365d0d5 from {username='slave'} at 127.0.0.1:41858
I20250625 20:01:11.644778 1082 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_c95f91afca114793990044d26cd96c64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:11.644309 1087 master_service.cc:948] ReplaceTablet: received request to replace tablet 2a388d1449bf455dbde985974c23e43f from {username='slave'} at 127.0.0.1:41858
I20250625 20:01:11.647179 2226 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Refusing update from remote peer 5852fb53503b438c92f294e0c44c916d: Log matching property violated. Preceding OpId in replica: term: 4 index: 84. Preceding OpId from leader: term: 5 index: 88. (index mismatch)
I20250625 20:01:11.648308 2286 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 83, Time since last communication: 0.000s
W20250625 20:01:11.650171 1075 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d -> Peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Couldn't send request to peer f87f9e18e5b948a4a3fb3666f106abd0. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20250625 20:01:11.651127 1326 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 6 out of 13 received events: NotificationEvent(eventId=22, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_063972c2ee4b4e468cf982177f09f91a, message={"server":"","servicePrincipal":"","db":"default","table":"table_063972c2ee4b4e468cf982177f09f91a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_063972c2ee4b4e468cf982177f09f91a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_063972c2ee4b4e468cf982177f09f91a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_063972c2ee4b4e468cf982177f09f91a\",\"kudu.table_id\":\"a054c42074394f1db21dfcf2c55d99b5\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_01c13560571f419e9b6f45edee3a2a37, message={"server":"","servicePrincipal":"","db":"default","table":"table_01c13560571f419e9b6f45edee3a2a37","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_01c13560571f419e9b6f45edee3a2a37\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_01c13560571f419e9b6f45edee3a2a37\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_01c13560571f419e9b6f45edee3a2a37\",\"kudu.table_id\":\"aee290f61c8f4d7b93d4cd7192eb9277\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_a31a093a024346aab31bc8d55a00253d, message={"server":"","servicePrincipal":"","db":"default","table":"table_a31a093a024346aab31bc8d55a00253d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a31a093a024346aab31bc8d55a00253d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_a31a093a024346aab31bc8d55a00253d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a31a093a024346aab31bc8d55a00253d\",\"kudu.table_id\":\"81aee385d1404c82aaeebee220cb9e68\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_60554f72b1aa48aa93aaea511a985fe7, message={"server":"","servicePrincipal":"","db":"default","table":"table_60554f72b1aa48aa93aaea511a985fe7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_60554f72b1aa48aa93aaea511a985fe7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881667},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_60554f72b1aa48aa93aaea511a985fe7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_60554f72b1aa48aa93aaea511a985fe7\",\"kudu.table_id\":\"e46f78618ca648e19c3935d0bf874a35\",\"transient_lastDdlTime\":\"1750881667\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250625 20:01:11.653111 2276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5852fb53503b438c92f294e0c44c916d. Latest consensus state: current_term: 5 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:11.653244 2276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f7058860dcd google::LogMessage::Fail() at ??:0
@ 0x7f7058864b93 google::LogMessage::SendToLog() at ??:0
I20250625 20:01:11.656899 2281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:11.657012 2281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:11.657429 2276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:11.657557 2276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:11.658322 2291 mvcc.cc:204] Tried to move back new op lower bound from 7171611327068868608 to 7171611326809927680. Current Snapshot: MvccSnapshot[applied={T|T < 7171611327068868608 or (T in {7171611327068868608})}]
@ 0x7f70588607cc google::LogMessage::Flush() at ??:0
I20250625 20:01:11.659364 1084 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Sending DeleteTablet for 3 replicas of tablet 21677e0e47df421c92f98ed580bbe7a7
I20250625 20:01:11.659766 1084 catalog_manager.cc:6652] ReplaceTablet: tablet 21677e0e47df421c92f98ed580bbe7a7 deleted and replaced by tablet e36a512748f54688b05f04d824af9ced
I20250625 20:01:11.659935 1397 tablet_service.cc:1515] Processing DeleteTablet for tablet 21677e0e47df421c92f98ed580bbe7a7 with delete_type TABLET_DATA_DELETED (replaced by tablet e36a512748f54688b05f04d824af9ced) from {username='slave'} at 127.0.0.1:46134
I20250625 20:01:11.660071 1084 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_bb2207463de346c3b1a4271f21eed795"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:11.660228 1537 tablet_service.cc:1515] Processing DeleteTablet for tablet 21677e0e47df421c92f98ed580bbe7a7 with delete_type TABLET_DATA_DELETED (replaced by tablet e36a512748f54688b05f04d824af9ced) from {username='slave'} at 127.0.0.1:57378
I20250625 20:01:11.660493 1674 tablet_service.cc:1515] Processing DeleteTablet for tablet 21677e0e47df421c92f98ed580bbe7a7 with delete_type TABLET_DATA_DELETED (replaced by tablet e36a512748f54688b05f04d824af9ced) from {username='slave'} at 127.0.0.1:43114
I20250625 20:01:11.660794 1087 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Sending DeleteTablet for 3 replicas of tablet 2a388d1449bf455dbde985974c23e43f
I20250625 20:01:11.660816 1083 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Sending DeleteTablet for 3 replicas of tablet 07686deb2e184c9fbae06fef5365d0d5
I20250625 20:01:11.660859 1090 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Sending DeleteTablet for 3 replicas of tablet b78b6f29890140de958cebed9868b303
I20250625 20:01:11.661077 1087 catalog_manager.cc:6652] ReplaceTablet: tablet 2a388d1449bf455dbde985974c23e43f deleted and replaced by tablet c00e8f8653bd42fda7bef627e99930cc
I20250625 20:01:11.661105 1083 catalog_manager.cc:6652] ReplaceTablet: tablet 07686deb2e184c9fbae06fef5365d0d5 deleted and replaced by tablet a2c3c78d11c146b9818adff5036da7c7
I20250625 20:01:11.661139 1090 catalog_manager.cc:6652] ReplaceTablet: tablet b78b6f29890140de958cebed9868b303 deleted and replaced by tablet d4cd08f25a694ab9949c119997cad26d
I20250625 20:01:11.661269 1087 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_117db1e3446d41099601e6b995a36d74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:11.661340 1090 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_ea337ae9a0e74819be0ba8940e1828e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:11.661507 1088 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41858:
table { table_name: "default.Table_01c13560571f419e9b6f45edee3a2a37" } modify_external_catalogs: true
I20250625 20:01:11.661597 1088 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41858:
table { table_name: "default.Table_01c13560571f419e9b6f45edee3a2a37" } modify_external_catalogs: true
I20250625 20:01:11.662156 1399 tablet_service.cc:1515] Processing DeleteTablet for tablet b78b6f29890140de958cebed9868b303 with delete_type TABLET_DATA_DELETED (replaced by tablet d4cd08f25a694ab9949c119997cad26d) from {username='slave'} at 127.0.0.1:46134
I20250625 20:01:11.662313 1400 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a388d1449bf455dbde985974c23e43f with delete_type TABLET_DATA_DELETED (replaced by tablet c00e8f8653bd42fda7bef627e99930cc) from {username='slave'} at 127.0.0.1:46134
I20250625 20:01:11.662880 1672 tablet_service.cc:1515] Processing DeleteTablet for tablet b78b6f29890140de958cebed9868b303 with delete_type TABLET_DATA_DELETED (replaced by tablet d4cd08f25a694ab9949c119997cad26d) from {username='slave'} at 127.0.0.1:43114
I20250625 20:01:11.663337 1535 tablet_service.cc:1515] Processing DeleteTablet for tablet 07686deb2e184c9fbae06fef5365d0d5 with delete_type TABLET_DATA_DELETED (replaced by tablet a2c3c78d11c146b9818adff5036da7c7) from {username='slave'} at 127.0.0.1:57378
I20250625 20:01:11.663354 1675 tablet_service.cc:1515] Processing DeleteTablet for tablet 07686deb2e184c9fbae06fef5365d0d5 with delete_type TABLET_DATA_DELETED (replaced by tablet a2c3c78d11c146b9818adff5036da7c7) from {username='slave'} at 127.0.0.1:43114
I20250625 20:01:11.663468 1541 tablet_service.cc:1515] Processing DeleteTablet for tablet b78b6f29890140de958cebed9868b303 with delete_type TABLET_DATA_DELETED (replaced by tablet d4cd08f25a694ab9949c119997cad26d) from {username='slave'} at 127.0.0.1:57378
I20250625 20:01:11.663456 1674 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a388d1449bf455dbde985974c23e43f with delete_type TABLET_DATA_DELETED (replaced by tablet c00e8f8653bd42fda7bef627e99930cc) from {username='slave'} at 127.0.0.1:43114
I20250625 20:01:11.663554 1537 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a388d1449bf455dbde985974c23e43f with delete_type TABLET_DATA_DELETED (replaced by tablet c00e8f8653bd42fda7bef627e99930cc) from {username='slave'} at 127.0.0.1:57378
@ 0x7f7058860a11 google::LogMessage::~LogMessage() at ??:0
I20250625 20:01:11.663666 1397 tablet_service.cc:1515] Processing DeleteTablet for tablet 07686deb2e184c9fbae06fef5365d0d5 with delete_type TABLET_DATA_DELETED (replaced by tablet a2c3c78d11c146b9818adff5036da7c7) from {username='slave'} at 127.0.0.1:46134
I20250625 20:01:11.661280 1083 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41858:
name: "default.Table_e91ea2924c1b40fc9d7610784c293467"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:11.664599 2303 tablet_replica.cc:331] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:11.664757 2303 raft_consensus.cc:2241] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:11.664875 2303 raft_consensus.cc:2270] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:11.665310 2303 ts_tablet_manager.cc:1905] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:11.665755 2301 tablet_replica.cc:331] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:11.665901 2301 raft_consensus.cc:2241] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:11.666095 2301 raft_consensus.cc:2270] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:11.666450 2301 ts_tablet_manager.cc:1905] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f705d7f9c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250625 20:01:11.667874 2303 ts_tablet_manager.cc:1918] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:11.667882 1537 tablet_service.cc:1468] Processing CreateTablet for tablet e36a512748f54688b05f04d824af9ced (DEFAULT_TABLE table=default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:11.667949 2303 log.cc:1199] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/21677e0e47df421c92f98ed580bbe7a7
I20250625 20:01:11.668180 1537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e36a512748f54688b05f04d824af9ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:11.668227 2303 ts_tablet_manager.cc:1939] T 21677e0e47df421c92f98ed580bbe7a7 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:11.668576 2301 ts_tablet_manager.cc:1918] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:11.668634 2301 log.cc:1199] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/21677e0e47df421c92f98ed580bbe7a7
I20250625 20:01:11.668830 2301 ts_tablet_manager.cc:1939] T 21677e0e47df421c92f98ed580bbe7a7 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:11.668886 2303 tablet_replica.cc:331] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:11.669015 2303 raft_consensus.cc:2241] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:11.669214 2303 raft_consensus.cc:2270] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:11.669521 2302 tablet_replica.cc:331] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:11.669602 2303 ts_tablet_manager.cc:1905] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:11.669631 2302 raft_consensus.cc:2241] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:11.669720 2302 raft_consensus.cc:2270] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:11.670037 2302 ts_tablet_manager.cc:1905] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:11.670622 1076 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet 21677e0e47df421c92f98ed580bbe7a7 (table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]) successfully deleted
I20250625 20:01:11.670836 1078 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet 21677e0e47df421c92f98ed580bbe7a7 (table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]) successfully deleted
I20250625 20:01:11.670951 1674 tablet_service.cc:1468] Processing CreateTablet for tablet e36a512748f54688b05f04d824af9ced (DEFAULT_TABLE table=default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:11.671190 1674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e36a512748f54688b05f04d824af9ced. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f705d7f8ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250625 20:01:11.671650 2303 ts_tablet_manager.cc:1918] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 20:01:11.671734 2303 log.cc:1199] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/b78b6f29890140de958cebed9868b303
I20250625 20:01:11.671988 2303 ts_tablet_manager.cc:1939] T b78b6f29890140de958cebed9868b303 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:11.672745 2303 tablet_replica.cc:331] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:11.672875 2303 raft_consensus.cc:2241] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:11.672992 2303 raft_consensus.cc:2270] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:11.673419 2303 ts_tablet_manager.cc:1905] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:11.673501 1076 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet b78b6f29890140de958cebed9868b303 (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]) successfully deleted
I20250625 20:01:11.674844 1397 tablet_service.cc:1468] Processing CreateTablet for tablet e36a512748f54688b05f04d824af9ced (DEFAULT_TABLE table=default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f705d7f830b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250625 20:01:11.675113 1397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e36a512748f54688b05f04d824af9ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:11.675235 2303 ts_tablet_manager.cc:1918] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 20:01:11.675300 2303 log.cc:1199] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/07686deb2e184c9fbae06fef5365d0d5
I20250625 20:01:11.675518 2302 ts_tablet_manager.cc:1918] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:11.675546 2303 ts_tablet_manager.cc:1939] T 07686deb2e184c9fbae06fef5365d0d5 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:11.675580 2302 log.cc:1199] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/21677e0e47df421c92f98ed580bbe7a7
I20250625 20:01:11.675815 2302 ts_tablet_manager.cc:1939] T 21677e0e47df421c92f98ed580bbe7a7 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:11.676226 1076 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet 07686deb2e184c9fbae06fef5365d0d5 (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]) successfully deleted
I20250625 20:01:11.676288 2303 tablet_replica.cc:331] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:11.676417 2303 raft_consensus.cc:2241] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:11.676514 2306 tablet_bootstrap.cc:492] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:11.676601 2303 raft_consensus.cc:2270] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:11.676964 2303 ts_tablet_manager.cc:1905] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:11.677582 2305 tablet_bootstrap.cc:492] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df: Bootstrap starting.
@ 0x7f705d7fda12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x562ae46d6752 std::function<>::operator()() at ??:0
I20250625 20:01:11.678815 2303 ts_tablet_manager.cc:1918] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:11.678881 2303 log.cc:1199] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/2a388d1449bf455dbde985974c23e43f
I20250625 20:01:11.679129 2303 ts_tablet_manager.cc:1939] T 2a388d1449bf455dbde985974c23e43f P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:11.679800 2301 tablet_replica.cc:331] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:11.679937 2301 raft_consensus.cc:2241] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:11.680063 2301 raft_consensus.cc:2270] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:11.680110 1076 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet 2a388d1449bf455dbde985974c23e43f (table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f]) successfully deleted
I20250625 20:01:11.680446 2301 ts_tablet_manager.cc:1905] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f7059cb20a1 kudu::Thread::SuperviseThread() at ??:0
I20250625 20:01:11.682492 2301 ts_tablet_manager.cc:1918] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 20:01:11.682560 2301 log.cc:1199] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/b78b6f29890140de958cebed9868b303
I20250625 20:01:11.682796 2301 ts_tablet_manager.cc:1939] T b78b6f29890140de958cebed9868b303 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:11.683223 1541 tablet_service.cc:1468] Processing CreateTablet for tablet a2c3c78d11c146b9818adff5036da7c7 (DEFAULT_TABLE table=default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:11.683410 2301 tablet_replica.cc:331] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:11.683471 1541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c3c78d11c146b9818adff5036da7c7. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f705a4a16db start_thread at ??:0
I20250625 20:01:11.683534 2301 raft_consensus.cc:2241] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:11.683633 2301 raft_consensus.cc:2270] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:11.684232 1675 tablet_service.cc:1468] Processing CreateTablet for tablet a2c3c78d11c146b9818adff5036da7c7 (DEFAULT_TABLE table=default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:11.684285 1674 tablet_service.cc:1468] Processing CreateTablet for tablet d4cd08f25a694ab9949c119997cad26d (DEFAULT_TABLE table=default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:11.684482 1675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c3c78d11c146b9818adff5036da7c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:11.684638 1672 tablet_service.cc:1468] Processing CreateTablet for tablet c00e8f8653bd42fda7bef627e99930cc (DEFAULT_TABLE table=default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:11.684839 1672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c00e8f8653bd42fda7bef627e99930cc. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f7057a1771f clone at ??:0
I20250625 20:01:11.685956 2302 tablet_replica.cc:331] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:11.686100 2302 raft_consensus.cc:2241] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:11.686331 2302 raft_consensus.cc:2270] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:11.686740 2302 ts_tablet_manager.cc:1905] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:11.687228 2306 tablet_bootstrap.cc:654] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:11.687252 2307 tablet_bootstrap.cc:492] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:11.687685 1674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4cd08f25a694ab9949c119997cad26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:11.687930 2305 tablet_bootstrap.cc:654] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:11.688264 2307 tablet_bootstrap.cc:654] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:11.690716 1537 tablet_service.cc:1468] Processing CreateTablet for tablet d4cd08f25a694ab9949c119997cad26d (DEFAULT_TABLE table=default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:11.690793 2302 ts_tablet_manager.cc:1918] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 20:01:11.690865 2302 log.cc:1199] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/07686deb2e184c9fbae06fef5365d0d5
I20250625 20:01:11.690945 1537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4cd08f25a694ab9949c119997cad26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:11.691134 2302 ts_tablet_manager.cc:1939] T 07686deb2e184c9fbae06fef5365d0d5 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:11.691782 2302 tablet_replica.cc:331] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:11.691910 2302 raft_consensus.cc:2241] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:11.692019 2302 raft_consensus.cc:2270] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:11.692410 2302 ts_tablet_manager.cc:1905] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:11.692900 1397 tablet_service.cc:1468] Processing CreateTablet for tablet d4cd08f25a694ab9949c119997cad26d (DEFAULT_TABLE table=default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:11.693091 1397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4cd08f25a694ab9949c119997cad26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:11.693094 2306 tablet_bootstrap.cc:492] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:11.693190 2306 ts_tablet_manager.cc:1397] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20250625 20:01:11.693723 2306 raft_consensus.cc:357] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.693835 2306 raft_consensus.cc:383] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:11.693869 2306 raft_consensus.cc:738] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:11.694008 2306 consensus_queue.cc:260] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.694329 2306 ts_tablet_manager.cc:1428] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 20:01:11.694730 2301 ts_tablet_manager.cc:1905] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:11.695523 2305 tablet_bootstrap.cc:492] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:11.695605 2305 ts_tablet_manager.cc:1397] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20250625 20:01:11.695888 2306 tablet_bootstrap.cc:492] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:11.696338 2305 raft_consensus.cc:357] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.696444 2305 raft_consensus.cc:383] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:11.696475 2305 raft_consensus.cc:738] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:11.696600 2305 consensus_queue.cc:260] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.696838 2306 tablet_bootstrap.cc:654] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:11.696892 2305 ts_tablet_manager.cc:1428] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:11.697079 2305 tablet_bootstrap.cc:492] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:11.698016 2305 tablet_bootstrap.cc:654] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:11.698972 2307 tablet_bootstrap.cc:492] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:11.699070 2307 ts_tablet_manager.cc:1397] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20250625 20:01:11.699574 2307 raft_consensus.cc:357] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.699745 2301 ts_tablet_manager.cc:1918] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:11.699800 2301 log.cc:1199] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/2a388d1449bf455dbde985974c23e43f
I20250625 20:01:11.699678 2307 raft_consensus.cc:383] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:11.699878 2307 raft_consensus.cc:738] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:11.700064 2301 ts_tablet_manager.cc:1939] T 2a388d1449bf455dbde985974c23e43f P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:11.700045 2307 consensus_queue.cc:260] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.700258 2302 ts_tablet_manager.cc:1918] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 20:01:11.700356 2302 log.cc:1199] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/b78b6f29890140de958cebed9868b303
I20250625 20:01:11.700605 2302 ts_tablet_manager.cc:1939] T b78b6f29890140de958cebed9868b303 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:11.700861 2307 ts_tablet_manager.cc:1428] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:11.701051 2307 tablet_bootstrap.cc:492] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:11.701355 2302 tablet_replica.cc:331] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:11.701496 2302 raft_consensus.cc:2241] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:11.701601 2302 raft_consensus.cc:2270] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:11.701960 2302 ts_tablet_manager.cc:1905] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:11.702641 2307 tablet_bootstrap.cc:654] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:11.704154 2302 ts_tablet_manager.cc:1918] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:11.704223 2302 log.cc:1199] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/2a388d1449bf455dbde985974c23e43f
I20250625 20:01:11.704480 2302 ts_tablet_manager.cc:1939] T 2a388d1449bf455dbde985974c23e43f P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:11.705207 2301 tablet_replica.cc:331] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:11.705317 2301 raft_consensus.cc:2241] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:11.705400 2301 raft_consensus.cc:2270] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:11.705734 2301 ts_tablet_manager.cc:1905] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:11.705826 2305 tablet_bootstrap.cc:492] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:11.705893 2305 ts_tablet_manager.cc:1397] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250625 20:01:11.705931 2307 tablet_bootstrap.cc:492] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:11.706013 2307 ts_tablet_manager.cc:1397] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20250625 20:01:11.706357 2305 raft_consensus.cc:357] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.706455 2305 raft_consensus.cc:383] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:11.706486 2305 raft_consensus.cc:738] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:11.706496 2307 raft_consensus.cc:357] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.706600 2307 raft_consensus.cc:383] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:11.706635 2307 raft_consensus.cc:738] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:11.706640 2305 consensus_queue.cc:260] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.707005 2305 ts_tablet_manager.cc:1428] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:11.707312 2305 tablet_bootstrap.cc:492] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:11.706801 2307 consensus_queue.cc:260] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.707649 2301 ts_tablet_manager.cc:1918] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 20:01:11.708441 2301 log.cc:1199] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/07686deb2e184c9fbae06fef5365d0d5
I20250625 20:01:11.708688 2301 ts_tablet_manager.cc:1939] T 07686deb2e184c9fbae06fef5365d0d5 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:11.708757 2305 tablet_bootstrap.cc:654] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:11.709228 2307 ts_tablet_manager.cc:1428] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 20:01:11.710454 2306 tablet_bootstrap.cc:492] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:11.710543 2306 ts_tablet_manager.cc:1397] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20250625 20:01:11.711107 2306 raft_consensus.cc:357] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.711218 2306 raft_consensus.cc:383] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:11.711248 2306 raft_consensus.cc:738] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:11.711390 2306 consensus_queue.cc:260] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.711645 2306 ts_tablet_manager.cc:1428] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:11.711874 2306 tablet_bootstrap.cc:492] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:11.712906 2306 tablet_bootstrap.cc:654] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:11.716085 2305 tablet_bootstrap.cc:492] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:11.716162 2305 ts_tablet_manager.cc:1397] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250625 20:01:11.716643 2305 raft_consensus.cc:357] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.716744 2305 raft_consensus.cc:383] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:11.716774 2305 raft_consensus.cc:738] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:11.716919 2305 consensus_queue.cc:260] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.717190 2305 ts_tablet_manager.cc:1428] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 20:01:11.717335 2306 tablet_bootstrap.cc:492] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:11.717422 2306 ts_tablet_manager.cc:1397] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250625 20:01:11.717880 2306 raft_consensus.cc:357] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.717976 2306 raft_consensus.cc:383] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:11.718007 2306 raft_consensus.cc:738] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:11.718104 2306 consensus_queue.cc:260] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.718374 2306 ts_tablet_manager.cc:1428] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:11.718567 2306 tablet_bootstrap.cc:492] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:11.719591 2306 tablet_bootstrap.cc:654] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:11.720980 2306 tablet_bootstrap.cc:492] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:11.721060 2306 ts_tablet_manager.cc:1397] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 20:01:11.721503 2306 raft_consensus.cc:357] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.721596 2306 raft_consensus.cc:383] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:11.721625 2306 raft_consensus.cc:738] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:11.721753 2306 consensus_queue.cc:260] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.722604 2306 ts_tablet_manager.cc:1428] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:11.803186 1850 raft_consensus.cc:491] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:11.803351 1850 raft_consensus.cc:513] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.803896 1850 leader_election.cc:290] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:11.804700 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4cd08f25a694ab9949c119997cad26d" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:11.804836 1701 raft_consensus.cc:2466] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:11.804811 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4cd08f25a694ab9949c119997cad26d" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:11.804931 1560 raft_consensus.cc:2466] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:11.805141 1358 leader_election.cc:304] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:11.805364 1850 raft_consensus.cc:2802] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:11.805416 1850 raft_consensus.cc:491] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:11.805450 1850 raft_consensus.cc:3058] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:11.805732 1935 raft_consensus.cc:491] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:11.805814 1935 raft_consensus.cc:513] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.806145 1935 leader_election.cc:290] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:11.806533 1850 raft_consensus.cc:513] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
W20250625 20:01:11.806697 1497 leader_election.cc:343] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: a2c3c78d11c146b9818adff5036da7c7
I20250625 20:01:11.806883 1850 leader_election.cc:290] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:11.807070 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2c3c78d11c146b9818adff5036da7c7" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:11.807178 1701 raft_consensus.cc:2466] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:11.807341 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4cd08f25a694ab9949c119997cad26d" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:11.807440 1495 leader_election.cc:304] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters: 27aa5b53b16747f9a72485aaf2417b08
I20250625 20:01:11.807468 1560 raft_consensus.cc:3058] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:11.807621 1935 raft_consensus.cc:2802] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:11.807655 1935 raft_consensus.cc:491] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:11.807690 1935 raft_consensus.cc:3058] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:11.808532 1560 raft_consensus.cc:2466] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:11.808611 1935 raft_consensus.cc:513] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.808841 1357 leader_election.cc:304] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:11.808914 1935 leader_election.cc:290] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:11.809177 1850 raft_consensus.cc:2802] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:11.809259 1850 raft_consensus.cc:695] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:11.809425 1850 consensus_queue.cc:237] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.809746 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4cd08f25a694ab9949c119997cad26d" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:11.809846 1701 raft_consensus.cc:3058] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:11.810729 1701 raft_consensus.cc:2466] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
W20250625 20:01:11.812281 1497 leader_election.cc:343] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: a2c3c78d11c146b9818adff5036da7c7
I20250625 20:01:11.812449 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2c3c78d11c146b9818adff5036da7c7" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:11.812577 1701 raft_consensus.cc:3058] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:11.813494 1701 raft_consensus.cc:2466] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:11.813797 1495 leader_election.cc:304] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters: 27aa5b53b16747f9a72485aaf2417b08
I20250625 20:01:11.813995 1935 raft_consensus.cc:2802] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:11.814174 1935 raft_consensus.cc:695] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Becoming Leader. State: Replica: c73846c0665d4d53a274d053946107df, State: Running, Role: LEADER
I20250625 20:01:11.814323 1935 consensus_queue.cc:237] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
W20250625 20:01:11.868450 1494 connection.cc:537] server connection from 127.0.0.1:57378 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:11.868620 1495 connection.cc:537] client connection to 127.0.204.62:36889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:11.868646 1755 connection.cc:537] client connection to 127.0.204.62:36889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:11.868628 1635 connection.cc:537] client connection to 127.0.204.62:36889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:11.869019 1755 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.204.62:36889)
W20250625 20:01:11.869385 1755 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.204.62:36889)
W20250625 20:01:11.869481 1755 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.204.62:36889)
W20250625 20:01:11.869889 1755 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.204.62:36889)
W20250625 20:01:11.870359 1360 connection.cc:537] server connection from 127.0.0.1:46134 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:11.870710 1636 connection.cc:537] server connection from 127.0.0.1:43114 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:11.870860 1358 connection.cc:537] client connection to 127.0.204.62:36889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:11.871500 2279 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 20:01:11.869255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.204.62:36889 (local address 127.0.204.3:53091)
0625 20:01:11.869469 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0625 20:01:11.869485 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0625 20:01:11.869581 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 20:01:11.870936 (+ 1355us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":129}
W20250625 20:01:11.871760 2153 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 20:01:11.870578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.204.62:36889 (local address 127.0.0.1:41878)
0625 20:01:11.871000 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0625 20:01:11.871056 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":338}
I20250625 20:01:11.991385 1843 raft_consensus.cc:491] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:11.991535 1843 raft_consensus.cc:513] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.991976 1843 leader_election.cc:290] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:11.992447 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e36a512748f54688b05f04d824af9ced" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:11.992447 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e36a512748f54688b05f04d824af9ced" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:11.992595 1424 raft_consensus.cc:2466] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:11.992595 1560 raft_consensus.cc:2466] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:11.992946 1637 leader_election.cc:304] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:11.993233 1843 raft_consensus.cc:2802] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:11.993297 1843 raft_consensus.cc:491] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:11.993338 1843 raft_consensus.cc:3058] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:11.994331 1843 raft_consensus.cc:513] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:11.994683 1843 leader_election.cc:290] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:11.994982 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e36a512748f54688b05f04d824af9ced" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:11.995080 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e36a512748f54688b05f04d824af9ced" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:11.995111 1424 raft_consensus.cc:3058] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:11.995193 1560 raft_consensus.cc:3058] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:11.996107 1560 raft_consensus.cc:2466] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:11.996107 1424 raft_consensus.cc:2466] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:11.996397 1634 leader_election.cc:304] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:11.996577 1843 raft_consensus.cc:2802] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:11.996889 1843 raft_consensus.cc:695] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
I20250625 20:01:11.997051 1843 consensus_queue.cc:237] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:12.099985 1843 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:12.100136 1843 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:12.100577 1843 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:01:12.101346 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:01:12.101495 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:01:12.101588 1637 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:01:12.101784 1843 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:01:12.182794 2335 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:12.182924 2335 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:12.183336 2335 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
W20250625 20:01:12.184412 2201 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:12.184506 2200 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:12.184582 2200 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9; no voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:12.184706 2335 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250625 20:01:12.264909 1850 consensus_queue.cc:1035] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:12.268503 1850 consensus_queue.cc:1035] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 20:01:12.311615 1497 consensus_peers.cc:487] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df -> Peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Couldn't send request to peer 27aa5b53b16747f9a72485aaf2417b08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2c3c78d11c146b9818adff5036da7c7. This is attempt 1: this message will repeat every 5th retry.
I20250625 20:01:12.416049 2330 consensus_queue.cc:1035] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:12.440593 1843 consensus_queue.cc:1035] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:12.444600 2350 consensus_queue.cc:1035] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:12.764801 2352 ts_tablet_manager.cc:927] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08: Initiating tablet copy from peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:12.765385 2352 tablet_copy_client.cc:323] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.2:43561
I20250625 20:01:12.766259 1580 tablet_copy_service.cc:140] P c73846c0665d4d53a274d053946107df: Received BeginTabletCopySession request for tablet a2c3c78d11c146b9818adff5036da7c7 from peer 27aa5b53b16747f9a72485aaf2417b08 ({username='slave'} at 127.0.204.1:43195)
I20250625 20:01:12.766426 1580 tablet_copy_service.cc:161] P c73846c0665d4d53a274d053946107df: Beginning new tablet copy session on tablet a2c3c78d11c146b9818adff5036da7c7 from peer 27aa5b53b16747f9a72485aaf2417b08 at {username='slave'} at 127.0.204.1:43195: session id = 27aa5b53b16747f9a72485aaf2417b08-a2c3c78d11c146b9818adff5036da7c7
I20250625 20:01:12.767675 1580 tablet_copy_source_session.cc:215] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 20:01:12.769340 2352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c3c78d11c146b9818adff5036da7c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:12.771348 2352 tablet_copy_client.cc:806] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08: tablet copy: Starting download of 0 data blocks...
I20250625 20:01:12.771553 2352 tablet_copy_client.cc:670] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08: tablet copy: Starting download of 1 WAL segments...
I20250625 20:01:12.773550 2352 tablet_copy_client.cc:538] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 20:01:12.774932 2352 tablet_bootstrap.cc:492] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:12.778493 2352 tablet_bootstrap.cc:492] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:01:12.778692 2352 tablet_bootstrap.cc:492] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap complete.
I20250625 20:01:12.778815 2352 ts_tablet_manager.cc:1397] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250625 20:01:12.779201 2352 raft_consensus.cc:357] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:12.779301 2352 raft_consensus.cc:738] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:12.779419 2352 consensus_queue.cc:260] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:12.779691 2352 ts_tablet_manager.cc:1428] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 20:01:12.780611 1580 tablet_copy_service.cc:342] P c73846c0665d4d53a274d053946107df: Request end of tablet copy session 27aa5b53b16747f9a72485aaf2417b08-a2c3c78d11c146b9818adff5036da7c7 received from {username='slave'} at 127.0.204.1:43195
I20250625 20:01:12.780758 1580 tablet_copy_service.cc:434] P c73846c0665d4d53a274d053946107df: ending tablet copy session 27aa5b53b16747f9a72485aaf2417b08-a2c3c78d11c146b9818adff5036da7c7 on tablet a2c3c78d11c146b9818adff5036da7c7 with peer 27aa5b53b16747f9a72485aaf2417b08
I20250625 20:01:12.873852 816 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu with pid 2182
I20250625 20:01:13.388975 1424 raft_consensus.cc:1215] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250625 20:01:13.853842 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.60:46191
--webserver_interface=127.0.204.60
--webserver_port=43997
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 20:01:13.961577 2371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:01:13.961822 2371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:13.961884 2371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:01:13.965700 2371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:01:13.965792 2371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:13.965816 2371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:01:13.965835 2371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:01:13.969885 2371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.60:46191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.204.60
--webserver_port=43997
--never_fsync=true
--heap_profile_path=/tmp/kudu.2371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:13.971047 2371 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:13.972528 2371 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:01:13.979552 2386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:13.979553 2388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:13.979635 2385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:13.979758 2371 server_base.cc:1048] running on GCE node
I20250625 20:01:13.980307 2371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:13.981249 2371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:13.982417 2371 hybrid_clock.cc:648] HybridClock initialized: now 1750881673982402 us; error 31 us; skew 500 ppm
I20250625 20:01:13.984490 2371 webserver.cc:469] Webserver started at http://127.0.204.60:43997/ using document root <none> and password file <none>
I20250625 20:01:13.985071 2371 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:13.985128 2371 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:13.988852 2371 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250625 20:01:13.990952 2394 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:13.992136 2371 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250625 20:01:13.992254 2371 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
uuid: "1fcdee0c0aac4013a5e3c77304b474d9"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
I20250625 20:01:13.992676 2371 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:14.002569 2371 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:14.003276 2371 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:14.003469 2371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:14.011143 2371 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.60:46191
I20250625 20:01:14.011157 2446 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.60:46191 every 8 connection(s)
I20250625 20:01:14.012369 2371 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/info.pb
I20250625 20:01:14.016286 2447 sys_catalog.cc:263] Verifying existing consensus state
I20250625 20:01:14.018250 2447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Bootstrap starting.
I20250625 20:01:14.019601 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 2371
I20250625 20:01:14.030860 2447 log.cc:826] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:14.076253 2447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=93 overwritten=0 applied=93 ignored=0} inserts{seen=62 ignored=0} mutations{seen=78 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:01:14.076661 2447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Bootstrap complete.
I20250625 20:01:14.079284 2447 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:14.079645 2447 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcdee0c0aac4013a5e3c77304b474d9, State: Initialized, Role: FOLLOWER
I20250625 20:01:14.080114 2447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 93, Last appended: 5.93, Last appended by leader: 93, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:14.080713 2451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:14.080857 2451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:14.081372 2447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:01:14.086272 2371 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:01:14.088749 2371 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:14.088755 2465 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:14.088938 2465 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: loading cluster ID for follower catalog manager: success
I20250625 20:01:14.092470 2465 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: acquiring CA information for follower catalog manager: success
I20250625 20:01:14.094774 2465 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 20:01:14.230124 2350 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:14.230286 2350 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:14.230787 2350 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:01:14.231564 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:01:14.231735 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:01:14.231827 1637 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:01:14.231969 2350 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:01:14.395314 1607 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:14.395812 1470 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:14.397326 2410 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" instance_seqno: 1750881665574847) as {username='slave'} at 127.0.204.1:49675; Asking this server to re-register.
I20250625 20:01:14.397332 2411 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c73846c0665d4d53a274d053946107df" instance_seqno: 1750881665791183) as {username='slave'} at 127.0.204.2:43135; Asking this server to re-register.
I20250625 20:01:14.398067 1470 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:14.398105 1607 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:14.399529 2411 ts_manager.cc:194] Registered new tserver with Master: 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:14.399600 2410 ts_manager.cc:194] Registered new tserver with Master: c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:14.433743 1747 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:14.434818 2411 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" instance_seqno: 1750881665979916) as {username='slave'} at 127.0.204.3:56043; Asking this server to re-register.
I20250625 20:01:14.435207 1747 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:14.436089 2411 ts_manager.cc:194] Registered new tserver with Master: 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:14.528115 2451 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:14.528352 2451 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:14.529644 2451 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
W20250625 20:01:14.530092 2396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:14.530891 2396 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:14.531025 2395 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:14.531100 2395 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9; no voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:14.531315 2451 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250625 20:01:15.061803 2481 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:15.061929 2481 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:15.062350 2481 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
W20250625 20:01:15.063416 2396 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:15.063534 2395 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:15.063619 2395 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9; no voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:15.063807 2481 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250625 20:01:15.846838 2489 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:15.846983 2489 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:15.847416 2489 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
W20250625 20:01:15.848529 2395 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
W20250625 20:01:15.848659 2396 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:15.848748 2396 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9; no voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:15.848877 2489 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250625 20:01:16.894157 2504 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:16.894284 2504 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:16.894709 2504 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
W20250625 20:01:16.895920 2396 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:16.896310 2395 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:16.896392 2395 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9; no voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:16.896519 2504 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250625 20:01:17.072438 2503 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:17.072572 2503 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:17.072997 2503 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:01:17.073730 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:01:17.073830 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:01:17.073891 1637 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:01:17.074015 2503 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:01:17.288755 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.61:38005
--webserver_interface=127.0.204.61
--webserver_port=44189
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 20:01:17.394176 2512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:01:17.394403 2512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:17.394456 2512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:01:17.398749 2512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:01:17.398883 2512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:17.398926 2512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:01:17.398952 2512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:01:17.402973 2512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.61:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.204.61
--webserver_port=44189
--never_fsync=true
--heap_profile_path=/tmp/kudu.2512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:17.404242 2512 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:17.405416 2512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:01:17.412020 2522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:17.412007 2521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:17.412003 2524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:17.412117 2512 server_base.cc:1048] running on GCE node
I20250625 20:01:17.412695 2512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:17.413573 2512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:17.414742 2512 hybrid_clock.cc:648] HybridClock initialized: now 1750881677414733 us; error 43 us; skew 500 ppm
I20250625 20:01:17.422533 2512 webserver.cc:469] Webserver started at http://127.0.204.61:44189/ using document root <none> and password file <none>
I20250625 20:01:17.423118 2512 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:17.423173 2512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:17.426766 2512 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250625 20:01:17.428948 2530 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:17.429914 2512 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:17.430007 2512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
uuid: "f87f9e18e5b948a4a3fb3666f106abd0"
format_stamp: "Formatted at 2025-06-25 20:01:04 on dist-test-slave-lc57"
I20250625 20:01:17.430357 2512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:17.459273 2512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:17.459957 2512 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:17.460103 2512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:17.466827 2512 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.61:38005
I20250625 20:01:17.466836 2582 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.61:38005 every 8 connection(s)
I20250625 20:01:17.467890 2512 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/info.pb
I20250625 20:01:17.471403 2583 sys_catalog.cc:263] Verifying existing consensus state
I20250625 20:01:17.472919 2583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Bootstrap starting.
I20250625 20:01:17.474778 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 2512
I20250625 20:01:17.483637 2583 log.cc:826] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:17.522725 2583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=86 overwritten=0 applied=86 ignored=0} inserts{seen=58 ignored=0} mutations{seen=70 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:01:17.523115 2583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Bootstrap complete.
I20250625 20:01:17.525727 2583 raft_consensus.cc:357] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:17.526079 2583 raft_consensus.cc:738] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87f9e18e5b948a4a3fb3666f106abd0, State: Initialized, Role: FOLLOWER
I20250625 20:01:17.526513 2583 consensus_queue.cc:260] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 86, Last appended: 4.86, Last appended by leader: 86, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:17.526968 2588 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:17.527076 2588 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:17.527544 2583 sys_catalog.cc:564] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:01:17.533316 2602 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:17.534710 2602 catalog_manager.cc:1554] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: loading cluster ID for follower catalog manager: success
I20250625 20:01:17.533344 2512 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:01:17.535404 2512 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:17.538543 2602 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: acquiring CA information for follower catalog manager: success
I20250625 20:01:17.541049 2602 catalog_manager.cc:1604] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 20:01:17.902288 1609 heartbeater.cc:344] Connected to a master server at 127.0.204.61:38005
I20250625 20:01:17.903255 1472 heartbeater.cc:344] Connected to a master server at 127.0.204.61:38005
I20250625 20:01:17.904512 2547 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c73846c0665d4d53a274d053946107df" instance_seqno: 1750881665791183) as {username='slave'} at 127.0.204.2:56629; Asking this server to re-register.
I20250625 20:01:17.904544 2546 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" instance_seqno: 1750881665574847) as {username='slave'} at 127.0.204.1:56039; Asking this server to re-register.
I20250625 20:01:17.905088 1609 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:17.905084 1472 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:17.906214 2546 ts_manager.cc:194] Registered new tserver with Master: c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:17.906309 2547 ts_manager.cc:194] Registered new tserver with Master: 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:17.940840 1749 heartbeater.cc:344] Connected to a master server at 127.0.204.61:38005
I20250625 20:01:17.942377 2547 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" instance_seqno: 1750881665979916) as {username='slave'} at 127.0.204.3:58979; Asking this server to re-register.
I20250625 20:01:17.942831 1749 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:17.943749 2547 ts_manager.cc:194] Registered new tserver with Master: 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:18.034708 2612 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:18.034904 2612 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:18.036178 2612 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:18.036756 2532 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:18.037667 2532 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:18.040047 2421 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" candidate_term: 5 candidate_status { last_received { term: 4 index: 86 } } ignore_live_leader: false dest_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" is_pre_election: true
I20250625 20:01:18.040342 2421 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f87f9e18e5b948a4a3fb3666f106abd0 in current term 5: Already voted for candidate 5852fb53503b438c92f294e0c44c916d in this term.
I20250625 20:01:18.040907 2534 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:18.041188 2612 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 4 FOLLOWER]: Advancing to term 5
I20250625 20:01:18.043147 2612 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 20:01:18.709349 2617 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:18.709486 2617 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:18.709969 2617 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:18.710453 2421 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" candidate_term: 6 candidate_status { last_received { term: 4 index: 86 } } ignore_live_leader: false dest_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" is_pre_election: true
I20250625 20:01:18.710659 2421 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f87f9e18e5b948a4a3fb3666f106abd0 for term 6 because replica has last-logged OpId of term: 5 index: 93, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 86.
W20250625 20:01:18.711143 2532 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:18.711215 2532 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:18.711333 2617 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250625 20:01:19.089861 2371 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:01:19.090019 2371 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:19.204726 2624 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:19.204867 2624 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:19.205344 2624 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
W20250625 20:01:19.206722 2396 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:19.209082 2557 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 93 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" is_pre_election: true
I20250625 20:01:19.209343 2557 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fcdee0c0aac4013a5e3c77304b474d9 in term 5.
I20250625 20:01:19.209780 2395 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0; no voters: 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:19.209971 2624 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250625 20:01:19.210037 2624 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:19.210079 2624 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 5 FOLLOWER]: Advancing to term 6
I20250625 20:01:19.212033 2624 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:19.212410 2624 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 election: Requested vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
I20250625 20:01:19.212821 2557 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 93 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0"
I20250625 20:01:19.212956 2557 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 5 FOLLOWER]: Advancing to term 6
W20250625 20:01:19.213832 2396 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:19.214118 2557 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcdee0c0aac4013a5e3c77304b474d9 in term 6.
I20250625 20:01:19.214386 2395 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0; no voters: 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:19.214545 2624 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 6 FOLLOWER]: Leader election won for term 6
I20250625 20:01:19.214757 2624 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 6 LEADER]: Becoming Leader. State: Replica: 1fcdee0c0aac4013a5e3c77304b474d9, State: Running, Role: LEADER
I20250625 20:01:19.215059 2624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 5.93, Last appended by leader: 93, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:19.216714 2628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fcdee0c0aac4013a5e3c77304b474d9. Latest consensus state: current_term: 6 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:19.216826 2628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:19.217098 2630 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:01:19.218753 2630 catalog_manager.cc:671] Loaded metadata for table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]
I20250625 20:01:19.219189 2630 catalog_manager.cc:671] Loaded metadata for table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]
I20250625 20:01:19.219388 2630 catalog_manager.cc:671] Loaded metadata for table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]
I20250625 20:01:19.219573 2630 catalog_manager.cc:671] Loaded metadata for table default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4]
I20250625 20:01:19.219787 2630 catalog_manager.cc:671] Loaded metadata for table default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]
I20250625 20:01:19.219976 2630 catalog_manager.cc:671] Loaded metadata for table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]
I20250625 20:01:19.220165 2630 catalog_manager.cc:671] Loaded metadata for table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996]
I20250625 20:01:19.220356 2630 catalog_manager.cc:671] Loaded metadata for table default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5]
I20250625 20:01:19.220541 2630 catalog_manager.cc:671] Loaded metadata for table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583]
I20250625 20:01:19.220734 2630 catalog_manager.cc:671] Loaded metadata for table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277]
I20250625 20:01:19.221025 2630 catalog_manager.cc:671] Loaded metadata for table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]
I20250625 20:01:19.221243 2630 catalog_manager.cc:671] Loaded metadata for table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f]
I20250625 20:01:19.221446 2630 catalog_manager.cc:671] Loaded metadata for table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]
I20250625 20:01:19.221668 2630 catalog_manager.cc:671] Loaded metadata for table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]
I20250625 20:01:19.221906 2630 catalog_manager.cc:671] Loaded metadata for table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]
I20250625 20:01:19.223134 2630 tablet_loader.cc:96] loaded metadata for tablet 06ef5241d21449169f228cfa37ff91b4 (table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996])
I20250625 20:01:19.223409 2630 tablet_loader.cc:96] loaded metadata for tablet 08697eef3f7f4c45ba3aacf36628e652 (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1])
I20250625 20:01:19.223948 2630 tablet_loader.cc:96] loaded metadata for tablet 4ec346c593cc448fbeb6d257ecc8dc52 (table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35])
I20250625 20:01:19.224151 2630 tablet_loader.cc:96] loaded metadata for tablet 6c8e829c96474e94a8de9e7e32b48350 (table default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4])
I20250625 20:01:19.224427 2630 tablet_loader.cc:96] loaded metadata for tablet 78631bcca66d4b16b344e57f8a1cc291 (table default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5])
I20250625 20:01:19.224798 2630 tablet_loader.cc:96] loaded metadata for tablet a2c3c78d11c146b9818adff5036da7c7 (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0])
I20250625 20:01:19.225167 2630 tablet_loader.cc:96] loaded metadata for tablet ba8f9c5ea37542f3b3f99a0c1d64b90d (table default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7])
I20250625 20:01:19.225301 2630 tablet_loader.cc:96] loaded metadata for tablet bb6c755fce6640d2a0b13a449d3adfb9 (table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583])
I20250625 20:01:19.225427 2630 tablet_loader.cc:96] loaded metadata for tablet c00e8f8653bd42fda7bef627e99930cc (table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f])
I20250625 20:01:19.225544 2630 tablet_loader.cc:96] loaded metadata for tablet c368f57ac08446e5a34b67f11ac51dc8 (table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e])
I20250625 20:01:19.225663 2630 tablet_loader.cc:96] loaded metadata for tablet c9a69c50e5f14364825e048fd19d4409 (table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa])
I20250625 20:01:19.225778 2630 tablet_loader.cc:96] loaded metadata for tablet cdb2a543898642d3abc3afa850178120 (table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277])
I20250625 20:01:19.225893 2630 tablet_loader.cc:96] loaded metadata for tablet d4cd08f25a694ab9949c119997cad26d (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809])
I20250625 20:01:19.226089 2630 tablet_loader.cc:96] loaded metadata for tablet d8dd4c0e80974a67b63293d83760fd8b (table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36])
I20250625 20:01:19.226290 2630 tablet_loader.cc:96] loaded metadata for tablet e36a512748f54688b05f04d824af9ced (table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8])
I20250625 20:01:19.226388 2630 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:01:19.227018 2630 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:19.227062 2630 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:01:19.227890 2630 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:01:19.228546 2630 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Loaded TSK: 0
I20250625 20:01:19.229204 2630 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:01:19.229813 2630 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 20:01:19.230352 2630 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 21
I20250625 20:01:19.316097 2411 master_service.cc:948] ReplaceTablet: received request to replace tablet e36a512748f54688b05f04d824af9ced from {username='slave'} at 127.0.0.1:45502
I20250625 20:01:19.316171 2410 master_service.cc:948] ReplaceTablet: received request to replace tablet c9a69c50e5f14364825e048fd19d4409 from {username='slave'} at 127.0.0.1:45502
I20250625 20:01:19.316689 2404 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45502:
table { table_name: "default.Table_fd28d4c5b48245c382a334b98a3066df" } modify_external_catalogs: true
I20250625 20:01:19.316735 2409 master_service.cc:948] ReplaceTablet: received request to replace tablet ba8f9c5ea37542f3b3f99a0c1d64b90d from {username='slave'} at 127.0.0.1:45502
I20250625 20:01:19.316854 2404 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45502:
table { table_name: "default.Table_fd28d4c5b48245c382a334b98a3066df" } modify_external_catalogs: true
I20250625 20:01:19.317526 2406 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45502:
table { table_name: "default.Table_1d0eb88cbdfd4b16bb66bfb3d502672b" } new_table_name: "default.Table_dd960e4b8a9c42309150252433773c76" modify_external_catalogs: true
I20250625 20:01:19.317811 2403 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45502:
table { table_name: "default.Table_d0f8eb84406f4d758e34b162746daa38" } new_table_name: "default.Table_a78f24eb6417409489ec4469005095b4" modify_external_catalogs: true
I20250625 20:01:19.318048 2405 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45502:
table { table_name: "default.Table_8302314b76364eadb44f02cc6d58a589" } new_table_name: "default.Table_2fcd1dbf63224757b5e1a9bad86e6a77" modify_external_catalogs: true
I20250625 20:01:19.318133 2408 master_service.cc:948] ReplaceTablet: received request to replace tablet 08697eef3f7f4c45ba3aacf36628e652 from {username='slave'} at 127.0.0.1:45502
I20250625 20:01:19.318213 2407 master_service.cc:948] ReplaceTablet: received request to replace tablet a2c3c78d11c146b9818adff5036da7c7 from {username='slave'} at 127.0.0.1:45502
I20250625 20:01:19.318599 2402 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45502:
table { table_name: "default.Table_030bc68e40784cf2a6bd261221a0f738" } modify_external_catalogs: true
I20250625 20:01:19.318702 2402 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45502:
table { table_name: "default.Table_030bc68e40784cf2a6bd261221a0f738" } modify_external_catalogs: true
I20250625 20:01:19.320562 2557 raft_consensus.cc:1273] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 6 FOLLOWER]: Refusing update from remote peer 1fcdee0c0aac4013a5e3c77304b474d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 86. Preceding OpId from leader: term: 6 index: 95. (index mismatch)
I20250625 20:01:19.323690 2628 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 86, Time since last communication: 0.000s
W20250625 20:01:19.324357 2396 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 -> Peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Couldn't send request to peer 5852fb53503b438c92f294e0c44c916d. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20250625 20:01:19.325114 2464 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 6 out of 13 received events: NotificationEvent(eventId=22, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_063972c2ee4b4e468cf982177f09f91a, message={"server":"","servicePrincipal":"","db":"default","table":"table_063972c2ee4b4e468cf982177f09f91a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_063972c2ee4b4e468cf982177f09f91a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_063972c2ee4b4e468cf982177f09f91a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_063972c2ee4b4e468cf982177f09f91a\",\"kudu.table_id\":\"a054c42074394f1db21dfcf2c55d99b5\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_01c13560571f419e9b6f45edee3a2a37, message={"server":"","servicePrincipal":"","db":"default","table":"table_01c13560571f419e9b6f45edee3a2a37","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_01c13560571f419e9b6f45edee3a2a37\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_01c13560571f419e9b6f45edee3a2a37\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_01c13560571f419e9b6f45edee3a2a37\",\"kudu.table_id\":\"aee290f61c8f4d7b93d4cd7192eb9277\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_a31a093a024346aab31bc8d55a00253d, message={"server":"","servicePrincipal":"","db":"default","table":"table_a31a093a024346aab31bc8d55a00253d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a31a093a024346aab31bc8d55a00253d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_a31a093a024346aab31bc8d55a00253d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a31a093a024346aab31bc8d55a00253d\",\"kudu.table_id\":\"81aee385d1404c82aaeebee220cb9e68\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_60554f72b1aa48aa93aaea511a985fe7, message={"server":"","servicePrincipal":"","db":"default","table":"table_60554f72b1aa48aa93aaea511a985fe7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_60554f72b1aa48aa93aaea511a985fe7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881667},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_60554f72b1aa48aa93aaea511a985fe7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_60554f72b1aa48aa93aaea511a985fe7\",\"kudu.table_id\":\"e46f78618ca648e19c3935d0bf874a35\",\"transient_lastDdlTime\":\"1750881667\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f75f7dc5dcd google::LogMessage::Fail() at ??:0
@ 0x7f75f7dc9b93 google::LogMessage::SendToLog() at ??:0
I20250625 20:01:19.330708 2632 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fcdee0c0aac4013a5e3c77304b474d9. Latest consensus state: current_term: 6 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:19.330832 2632 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:19.330919 2632 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:19.330976 2632 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f75f7dc57cc google::LogMessage::Flush() at ??:0
I20250625 20:01:19.332744 2628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:19.332863 2628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:19.332985 2631 mvcc.cc:204] Tried to move back new op lower bound from 7171611358504660992 to 7171611358067081216. Current Snapshot: MvccSnapshot[applied={T|T < 7171611358491045888}]
@ 0x7f75f7dc5a11 google::LogMessage::~LogMessage() at ??:0
I20250625 20:01:19.333817 2632 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:19.333914 2632 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:19.334944 2633 mvcc.cc:204] Tried to move back new op lower bound from 7171611358504660992 to 7171611326809927680. Current Snapshot: MvccSnapshot[applied={T|T < 7171611358501482496}]
I20250625 20:01:19.336246 2407 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Sending DeleteTablet for 3 replicas of tablet a2c3c78d11c146b9818adff5036da7c7
I20250625 20:01:19.336715 2410 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Sending DeleteTablet for 3 replicas of tablet c9a69c50e5f14364825e048fd19d4409
I20250625 20:01:19.336982 2408 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Sending DeleteTablet for 3 replicas of tablet 08697eef3f7f4c45ba3aacf36628e652
I20250625 20:01:19.337237 2409 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Sending DeleteTablet for 3 replicas of tablet ba8f9c5ea37542f3b3f99a0c1d64b90d
I20250625 20:01:19.336999 2411 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Sending DeleteTablet for 3 replicas of tablet e36a512748f54688b05f04d824af9ced
I20250625 20:01:19.337484 2407 catalog_manager.cc:6652] ReplaceTablet: tablet a2c3c78d11c146b9818adff5036da7c7 deleted and replaced by tablet e44908760f6245a282f72cec3b5773fe
I20250625 20:01:19.337517 2411 catalog_manager.cc:6652] ReplaceTablet: tablet e36a512748f54688b05f04d824af9ced deleted and replaced by tablet d55b31ba2d2f474ca4871ee6049f24bf
I20250625 20:01:19.337690 2409 catalog_manager.cc:6652] ReplaceTablet: tablet ba8f9c5ea37542f3b3f99a0c1d64b90d deleted and replaced by tablet 7b1d8f876f744c868857b56e49dda3e7
@ 0x7f75fcd5ec91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250625 20:01:19.337761 2407 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45502:
table { table_name: "default.Table_ef511cc05ad8422e98275bbb9b82ecb4" } modify_external_catalogs: true
I20250625 20:01:19.337788 2411 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45502:
table { table_name: "default.Table_976af21482a74bc79eb800684cce978e" } new_table_name: "default.Table_ad0e628a3e004a9da4a42cb7d6051e3e" modify_external_catalogs: true
I20250625 20:01:19.337872 2407 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45502:
table { table_name: "default.Table_ef511cc05ad8422e98275bbb9b82ecb4" } modify_external_catalogs: true
I20250625 20:01:19.338186 2408 catalog_manager.cc:6652] ReplaceTablet: tablet 08697eef3f7f4c45ba3aacf36628e652 deleted and replaced by tablet eb717c706f3d428bbd5bf6c8e4f839aa
I20250625 20:01:19.338342 2408 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45502:
table { table_name: "default.Table_063972c2ee4b4e468cf982177f09f91a" } new_table_name: "default.Table_980a5f7891e24ccc87c72a858cc5076f" modify_external_catalogs: true
@ 0x7f75fcd5dba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250625 20:01:19.337810 2409 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45502:
table { table_name: "default.Table_60554f72b1aa48aa93aaea511a985fe7" } modify_external_catalogs: true
I20250625 20:01:19.342397 2409 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45502:
table { table_name: "default.Table_60554f72b1aa48aa93aaea511a985fe7" } modify_external_catalogs: true
I20250625 20:01:19.342628 2410 catalog_manager.cc:6652] ReplaceTablet: tablet c9a69c50e5f14364825e048fd19d4409 deleted and replaced by tablet 01c760524ea849bcbae35ed5edde6858
I20250625 20:01:19.342941 2410 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45502:
name: "default.Table_3629c083428e4ae0a977ffb16af83bc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f75fcd5d30b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250625 20:01:19.345685 1397 tablet_service.cc:1515] Processing DeleteTablet for tablet a2c3c78d11c146b9818adff5036da7c7 with delete_type TABLET_DATA_DELETED (replaced by tablet e44908760f6245a282f72cec3b5773fe) from {username='slave'} at 127.0.0.1:54590
I20250625 20:01:19.345886 1400 tablet_service.cc:1515] Processing DeleteTablet for tablet c9a69c50e5f14364825e048fd19d4409 with delete_type TABLET_DATA_DELETED (replaced by tablet 01c760524ea849bcbae35ed5edde6858) from {username='slave'} at 127.0.0.1:54590
I20250625 20:01:19.346025 1398 tablet_service.cc:1515] Processing DeleteTablet for tablet e36a512748f54688b05f04d824af9ced with delete_type TABLET_DATA_DELETED (replaced by tablet d55b31ba2d2f474ca4871ee6049f24bf) from {username='slave'} at 127.0.0.1:54590
I20250625 20:01:19.346153 1401 tablet_service.cc:1515] Processing DeleteTablet for tablet ba8f9c5ea37542f3b3f99a0c1d64b90d with delete_type TABLET_DATA_DELETED (replaced by tablet 7b1d8f876f744c868857b56e49dda3e7) from {username='slave'} at 127.0.0.1:54590
I20250625 20:01:19.345902 1399 tablet_service.cc:1515] Processing DeleteTablet for tablet 08697eef3f7f4c45ba3aacf36628e652 with delete_type TABLET_DATA_DELETED (replaced by tablet eb717c706f3d428bbd5bf6c8e4f839aa) from {username='slave'} at 127.0.0.1:54590
@ 0x7f75fcd62a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250625 20:01:19.348186 1674 tablet_service.cc:1515] Processing DeleteTablet for tablet a2c3c78d11c146b9818adff5036da7c7 with delete_type TABLET_DATA_DELETED (replaced by tablet e44908760f6245a282f72cec3b5773fe) from {username='slave'} at 127.0.0.1:43822
I20250625 20:01:19.348438 1672 tablet_service.cc:1515] Processing DeleteTablet for tablet e36a512748f54688b05f04d824af9ced with delete_type TABLET_DATA_DELETED (replaced by tablet d55b31ba2d2f474ca4871ee6049f24bf) from {username='slave'} at 127.0.0.1:43822
@ 0x556466e32752 std::function<>::operator()() at ??:0
I20250625 20:01:19.348822 1675 tablet_service.cc:1515] Processing DeleteTablet for tablet ba8f9c5ea37542f3b3f99a0c1d64b90d with delete_type TABLET_DATA_DELETED (replaced by tablet 7b1d8f876f744c868857b56e49dda3e7) from {username='slave'} at 127.0.0.1:43822
I20250625 20:01:19.348949 1673 tablet_service.cc:1515] Processing DeleteTablet for tablet 08697eef3f7f4c45ba3aacf36628e652 with delete_type TABLET_DATA_DELETED (replaced by tablet eb717c706f3d428bbd5bf6c8e4f839aa) from {username='slave'} at 127.0.0.1:43822
I20250625 20:01:19.349069 1676 tablet_service.cc:1515] Processing DeleteTablet for tablet c9a69c50e5f14364825e048fd19d4409 with delete_type TABLET_DATA_DELETED (replaced by tablet 01c760524ea849bcbae35ed5edde6858) from {username='slave'} at 127.0.0.1:43822
I20250625 20:01:19.349895 1674 tablet_service.cc:1468] Processing CreateTablet for tablet e44908760f6245a282f72cec3b5773fe (DEFAULT_TABLE table=default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:19.350179 1674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44908760f6245a282f72cec3b5773fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:19.350997 2652 tablet_replica.cc:331] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:19.351130 2652 raft_consensus.cc:2241] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:19.351243 2652 raft_consensus.cc:2270] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f75f92170a1 kudu::Thread::SuperviseThread() at ??:0
I20250625 20:01:19.351619 2652 ts_tablet_manager.cc:1905] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:19.352631 1676 tablet_service.cc:1468] Processing CreateTablet for tablet eb717c706f3d428bbd5bf6c8e4f839aa (DEFAULT_TABLE table=default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:19.352851 1676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb717c706f3d428bbd5bf6c8e4f839aa. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f75f9a066db start_thread at ??:0
I20250625 20:01:19.354168 1537 tablet_service.cc:1515] Processing DeleteTablet for tablet a2c3c78d11c146b9818adff5036da7c7 with delete_type TABLET_DATA_DELETED (replaced by tablet e44908760f6245a282f72cec3b5773fe) from {username='slave'} at 127.0.0.1:52290
I20250625 20:01:19.354218 2652 ts_tablet_manager.cc:1918] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:19.354280 2652 log.cc:1199] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/a2c3c78d11c146b9818adff5036da7c7
I20250625 20:01:19.354519 2652 ts_tablet_manager.cc:1939] T a2c3c78d11c146b9818adff5036da7c7 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:19.354665 1541 tablet_service.cc:1515] Processing DeleteTablet for tablet c9a69c50e5f14364825e048fd19d4409 with delete_type TABLET_DATA_DELETED (replaced by tablet 01c760524ea849bcbae35ed5edde6858) from {username='slave'} at 127.0.0.1:52290
I20250625 20:01:19.354796 1535 tablet_service.cc:1515] Processing DeleteTablet for tablet e36a512748f54688b05f04d824af9ced with delete_type TABLET_DATA_DELETED (replaced by tablet d55b31ba2d2f474ca4871ee6049f24bf) from {username='slave'} at 127.0.0.1:52290
I20250625 20:01:19.354763 1538 tablet_service.cc:1468] Processing CreateTablet for tablet 7b1d8f876f744c868857b56e49dda3e7 (DEFAULT_TABLE table=default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:19.354852 1673 tablet_service.cc:1468] Processing CreateTablet for tablet d55b31ba2d2f474ca4871ee6049f24bf (DEFAULT_TABLE table=default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:19.354894 1533 tablet_service.cc:1515] Processing DeleteTablet for tablet ba8f9c5ea37542f3b3f99a0c1d64b90d with delete_type TABLET_DATA_DELETED (replaced by tablet 7b1d8f876f744c868857b56e49dda3e7) from {username='slave'} at 127.0.0.1:52290
I20250625 20:01:19.355020 2652 tablet_replica.cc:331] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:19.355017 1532 tablet_service.cc:1515] Processing DeleteTablet for tablet 08697eef3f7f4c45ba3aacf36628e652 with delete_type TABLET_DATA_DELETED (replaced by tablet eb717c706f3d428bbd5bf6c8e4f839aa) from {username='slave'} at 127.0.0.1:52290
I20250625 20:01:19.355020 1538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b1d8f876f744c868857b56e49dda3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:19.355073 1673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d55b31ba2d2f474ca4871ee6049f24bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:19.355131 2652 raft_consensus.cc:2241] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:19.355270 1534 tablet_service.cc:1468] Processing CreateTablet for tablet eb717c706f3d428bbd5bf6c8e4f839aa (DEFAULT_TABLE table=default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:19.355312 2652 raft_consensus.cc:2270] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:19.355484 1534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb717c706f3d428bbd5bf6c8e4f839aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:19.355652 2652 ts_tablet_manager.cc:1905] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:19.357005 1672 tablet_service.cc:1468] Processing CreateTablet for tablet 7b1d8f876f744c868857b56e49dda3e7 (DEFAULT_TABLE table=default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:19.357213 1672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b1d8f876f744c868857b56e49dda3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:19.357373 2652 ts_tablet_manager.cc:1918] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:19.357429 2652 log.cc:1199] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/e36a512748f54688b05f04d824af9ced
I20250625 20:01:19.357452 1536 tablet_service.cc:1468] Processing CreateTablet for tablet 01c760524ea849bcbae35ed5edde6858 (DEFAULT_TABLE table=default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:19.357641 1536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01c760524ea849bcbae35ed5edde6858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:19.357703 2652 ts_tablet_manager.cc:1939] T e36a512748f54688b05f04d824af9ced P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:19.358101 2652 tablet_replica.cc:331] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:19.358186 2652 raft_consensus.cc:2241] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:19.358314 2652 raft_consensus.cc:2270] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:19.358563 2652 ts_tablet_manager.cc:1905] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:19.359087 1675 tablet_service.cc:1468] Processing CreateTablet for tablet 01c760524ea849bcbae35ed5edde6858 (DEFAULT_TABLE table=default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:19.359272 1675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01c760524ea849bcbae35ed5edde6858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:19.359477 1539 tablet_service.cc:1468] Processing CreateTablet for tablet e44908760f6245a282f72cec3b5773fe (DEFAULT_TABLE table=default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:19.359683 1539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44908760f6245a282f72cec3b5773fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:19.359758 2652 ts_tablet_manager.cc:1918] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:19.359815 2652 log.cc:1199] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/ba8f9c5ea37542f3b3f99a0c1d64b90d
I20250625 20:01:19.360024 2652 ts_tablet_manager.cc:1939] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:19.360330 1537 tablet_service.cc:1468] Processing CreateTablet for tablet d55b31ba2d2f474ca4871ee6049f24bf (DEFAULT_TABLE table=default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:19.360518 1537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d55b31ba2d2f474ca4871ee6049f24bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:19.360565 2652 tablet_replica.cc:331] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:19.360673 2652 raft_consensus.cc:2241] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:19.360828 2652 raft_consensus.cc:2270] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:19.361140 2652 ts_tablet_manager.cc:1905] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:19.361716 1401 tablet_service.cc:1468] Processing CreateTablet for tablet d55b31ba2d2f474ca4871ee6049f24bf (DEFAULT_TABLE table=default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:19.361779 1400 tablet_service.cc:1468] Processing CreateTablet for tablet 7b1d8f876f744c868857b56e49dda3e7 (DEFAULT_TABLE table=default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:19.361933 1397 tablet_service.cc:1468] Processing CreateTablet for tablet e44908760f6245a282f72cec3b5773fe (DEFAULT_TABLE table=default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:19.361992 1401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d55b31ba2d2f474ca4871ee6049f24bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:19.362110 1400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b1d8f876f744c868857b56e49dda3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:19.361790 1398 tablet_service.cc:1468] Processing CreateTablet for tablet 01c760524ea849bcbae35ed5edde6858 (DEFAULT_TABLE table=default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:19.362283 1398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01c760524ea849bcbae35ed5edde6858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:19.362331 1399 tablet_service.cc:1468] Processing CreateTablet for tablet eb717c706f3d428bbd5bf6c8e4f839aa (DEFAULT_TABLE table=default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:19.362495 1399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb717c706f3d428bbd5bf6c8e4f839aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:19.364228 1397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44908760f6245a282f72cec3b5773fe. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f75f6f7c71f clone at ??:0
I20250625 20:01:19.365449 2656 tablet_bootstrap.cc:492] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:19.365777 2652 ts_tablet_manager.cc:1918] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:19.365839 2652 log.cc:1199] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/08697eef3f7f4c45ba3aacf36628e652
I20250625 20:01:19.366068 2652 ts_tablet_manager.cc:1939] T 08697eef3f7f4c45ba3aacf36628e652 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:19.366192 2653 tablet_bootstrap.cc:492] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:19.366614 2656 tablet_bootstrap.cc:654] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:19.366660 2652 tablet_replica.cc:331] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:19.366757 2652 raft_consensus.cc:2241] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:19.367111 2653 tablet_bootstrap.cc:654] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:19.367390 2655 tablet_bootstrap.cc:492] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:19.367694 2651 tablet_replica.cc:331] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:19.367828 2651 raft_consensus.cc:2241] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:19.367900 2656 tablet_bootstrap.cc:492] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:19.367914 2651 raft_consensus.cc:2270] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:19.366847 2652 raft_consensus.cc:2270] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:19.367962 2656 ts_tablet_manager.cc:1397] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 20:01:19.368214 2651 ts_tablet_manager.cc:1905] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:19.368263 2652 ts_tablet_manager.cc:1905] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:19.368331 2655 tablet_bootstrap.cc:654] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:19.368381 2656 raft_consensus.cc:357] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.368471 2656 raft_consensus.cc:383] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:19.368499 2656 raft_consensus.cc:738] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:19.368631 2656 consensus_queue.cc:260] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.369051 2654 tablet_replica.cc:331] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:19.369164 2654 raft_consensus.cc:2241] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:19.369385 2654 raft_consensus.cc:2270] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:19.369709 2654 ts_tablet_manager.cc:1905] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:19.369974 2652 ts_tablet_manager.cc:1918] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:19.370008 2651 ts_tablet_manager.cc:1918] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:19.370038 2652 log.cc:1199] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/c9a69c50e5f14364825e048fd19d4409
I20250625 20:01:19.370065 2651 log.cc:1199] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/a2c3c78d11c146b9818adff5036da7c7
I20250625 20:01:19.370244 2652 ts_tablet_manager.cc:1939] T c9a69c50e5f14364825e048fd19d4409 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:19.370607 2651 ts_tablet_manager.cc:1939] T a2c3c78d11c146b9818adff5036da7c7 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:19.370697 2656 ts_tablet_manager.cc:1428] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 20:01:19.371169 2651 tablet_replica.cc:331] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:19.371269 2651 raft_consensus.cc:2241] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:19.371431 2651 raft_consensus.cc:2270] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:19.371551 2654 ts_tablet_manager.cc:1918] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:19.371608 2654 log.cc:1199] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/a2c3c78d11c146b9818adff5036da7c7
I20250625 20:01:19.371850 2654 ts_tablet_manager.cc:1939] T a2c3c78d11c146b9818adff5036da7c7 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:19.372049 2656 tablet_bootstrap.cc:492] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:19.372413 2654 tablet_replica.cc:331] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:19.372509 2654 raft_consensus.cc:2241] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:19.372586 2654 raft_consensus.cc:2270] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:19.372833 2654 ts_tablet_manager.cc:1905] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:19.372952 2656 tablet_bootstrap.cc:654] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:19.374130 2654 ts_tablet_manager.cc:1918] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:19.374174 2654 log.cc:1199] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/c9a69c50e5f14364825e048fd19d4409
I20250625 20:01:19.374217 2653 tablet_bootstrap.cc:492] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:19.374305 2653 ts_tablet_manager.cc:1397] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250625 20:01:19.374383 2654 ts_tablet_manager.cc:1939] T c9a69c50e5f14364825e048fd19d4409 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:19.374797 2654 tablet_replica.cc:331] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:19.374752 2653 raft_consensus.cc:357] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.374855 2653 raft_consensus.cc:383] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:19.374886 2653 raft_consensus.cc:738] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:19.374893 2654 raft_consensus.cc:2241] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:19.374961 2654 raft_consensus.cc:2270] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:19.375010 2653 consensus_queue.cc:260] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.375245 2654 ts_tablet_manager.cc:1905] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:19.375281 2653 ts_tablet_manager.cc:1428] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 20:01:19.375445 2653 tablet_bootstrap.cc:492] T eb717c706f3d428bbd5bf6c8e4f839aa P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:19.376135 2656 tablet_bootstrap.cc:492] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:19.376210 2656 ts_tablet_manager.cc:1397] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250625 20:01:19.376645 2656 raft_consensus.cc:357] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.376739 2656 raft_consensus.cc:383] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:19.376768 2656 raft_consensus.cc:738] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:19.376927 2654 ts_tablet_manager.cc:1918] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:19.376986 2654 log.cc:1199] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/e36a512748f54688b05f04d824af9ced
I20250625 20:01:19.377190 2654 ts_tablet_manager.cc:1939] T e36a512748f54688b05f04d824af9ced P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:19.377599 2653 tablet_bootstrap.cc:654] T eb717c706f3d428bbd5bf6c8e4f839aa P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:19.378022 2654 tablet_replica.cc:331] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:19.378129 2654 raft_consensus.cc:2241] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:19.378213 2654 raft_consensus.cc:2270] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:19.378194 2656 consensus_queue.cc:260] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.378518 2654 ts_tablet_manager.cc:1905] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:19.378679 2656 ts_tablet_manager.cc:1428] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:19.379381 2653 tablet_bootstrap.cc:492] T eb717c706f3d428bbd5bf6c8e4f839aa P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:19.379459 2653 ts_tablet_manager.cc:1397] T eb717c706f3d428bbd5bf6c8e4f839aa P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250625 20:01:19.380045 2654 ts_tablet_manager.cc:1918] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:19.379978 2653 raft_consensus.cc:357] T eb717c706f3d428bbd5bf6c8e4f839aa P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.380090 2654 log.cc:1199] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/ba8f9c5ea37542f3b3f99a0c1d64b90d
I20250625 20:01:19.380095 2653 raft_consensus.cc:383] T eb717c706f3d428bbd5bf6c8e4f839aa P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:19.380126 2653 raft_consensus.cc:738] T eb717c706f3d428bbd5bf6c8e4f839aa P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:19.380249 2654 ts_tablet_manager.cc:1939] T ba8f9c5ea37542f3b3f99a0c1d64b90d P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:19.380247 2653 consensus_queue.cc:260] T eb717c706f3d428bbd5bf6c8e4f839aa P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.380509 2653 ts_tablet_manager.cc:1428] T eb717c706f3d428bbd5bf6c8e4f839aa P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:19.380673 2653 tablet_bootstrap.cc:492] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:19.380811 2654 tablet_replica.cc:331] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:19.380908 2654 raft_consensus.cc:2241] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:19.380991 2654 raft_consensus.cc:2270] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:19.381383 2654 ts_tablet_manager.cc:1905] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:19.381717 2653 tablet_bootstrap.cc:654] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:19.383000 2655 tablet_bootstrap.cc:492] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:19.383088 2655 ts_tablet_manager.cc:1397] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250625 20:01:19.383258 2654 ts_tablet_manager.cc:1918] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:19.383349 2654 log.cc:1199] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/08697eef3f7f4c45ba3aacf36628e652
I20250625 20:01:19.383584 2655 raft_consensus.cc:357] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.383677 2655 raft_consensus.cc:383] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:19.383785 2654 ts_tablet_manager.cc:1939] T 08697eef3f7f4c45ba3aacf36628e652 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:19.384917 2655 raft_consensus.cc:738] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:19.385082 2655 consensus_queue.cc:260] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.385391 2655 ts_tablet_manager.cc:1428] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:19.385596 2655 tablet_bootstrap.cc:492] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:19.386662 2655 tablet_bootstrap.cc:654] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:19.386760 2656 tablet_bootstrap.cc:492] T eb717c706f3d428bbd5bf6c8e4f839aa P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:19.387768 2656 tablet_bootstrap.cc:654] T eb717c706f3d428bbd5bf6c8e4f839aa P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:19.388271 2651 ts_tablet_manager.cc:1905] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:19.389437 2656 tablet_bootstrap.cc:492] T eb717c706f3d428bbd5bf6c8e4f839aa P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:19.389509 2656 ts_tablet_manager.cc:1397] T eb717c706f3d428bbd5bf6c8e4f839aa P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250625 20:01:19.389704 2655 tablet_bootstrap.cc:492] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:19.389780 2655 ts_tablet_manager.cc:1397] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250625 20:01:19.389955 2656 raft_consensus.cc:357] T eb717c706f3d428bbd5bf6c8e4f839aa P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.390064 2656 raft_consensus.cc:383] T eb717c706f3d428bbd5bf6c8e4f839aa P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:19.390095 2656 raft_consensus.cc:738] T eb717c706f3d428bbd5bf6c8e4f839aa P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:19.390218 2656 consensus_queue.cc:260] T eb717c706f3d428bbd5bf6c8e4f839aa P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.390244 2655 raft_consensus.cc:357] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.390362 2655 raft_consensus.cc:383] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:19.390390 2655 raft_consensus.cc:738] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:19.390506 2655 consensus_queue.cc:260] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.390791 2655 ts_tablet_manager.cc:1428] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:19.390965 2655 tablet_bootstrap.cc:492] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:19.390385 2651 ts_tablet_manager.cc:1918] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:19.391054 2651 log.cc:1199] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/c9a69c50e5f14364825e048fd19d4409
I20250625 20:01:19.391292 2651 ts_tablet_manager.cc:1939] T c9a69c50e5f14364825e048fd19d4409 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:19.391873 2651 tablet_replica.cc:331] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:19.392031 2656 ts_tablet_manager.cc:1428] T eb717c706f3d428bbd5bf6c8e4f839aa P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 20:01:19.392766 2651 raft_consensus.cc:2241] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:19.392868 2651 raft_consensus.cc:2270] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:19.393210 2651 ts_tablet_manager.cc:1905] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:19.393525 2656 tablet_bootstrap.cc:492] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:19.393963 2655 tablet_bootstrap.cc:654] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:19.394408 2656 tablet_bootstrap.cc:654] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:19.395044 2655 tablet_bootstrap.cc:492] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:19.395128 2655 ts_tablet_manager.cc:1397] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250625 20:01:19.395145 2651 ts_tablet_manager.cc:1918] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:19.395190 2651 log.cc:1199] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/e36a512748f54688b05f04d824af9ced
I20250625 20:01:19.395350 2651 ts_tablet_manager.cc:1939] T e36a512748f54688b05f04d824af9ced P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:19.395639 2655 raft_consensus.cc:357] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.395776 2655 raft_consensus.cc:383] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:19.395812 2655 raft_consensus.cc:738] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:19.395956 2655 consensus_queue.cc:260] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.396167 2651 tablet_replica.cc:331] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:19.396240 2653 tablet_bootstrap.cc:492] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:19.396273 2651 raft_consensus.cc:2241] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:19.396296 2656 tablet_bootstrap.cc:492] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:19.396330 2653 ts_tablet_manager.cc:1397] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250625 20:01:19.396355 2656 ts_tablet_manager.cc:1397] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250625 20:01:19.396361 2651 raft_consensus.cc:2270] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:19.396674 2651 ts_tablet_manager.cc:1905] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:19.396764 2656 raft_consensus.cc:357] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.396780 2653 raft_consensus.cc:357] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.396847 2656 raft_consensus.cc:383] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:19.396875 2656 raft_consensus.cc:738] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:19.396879 2653 raft_consensus.cc:383] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:19.396912 2653 raft_consensus.cc:738] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:19.397002 2656 consensus_queue.cc:260] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.397068 2653 consensus_queue.cc:260] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.398396 2651 ts_tablet_manager.cc:1918] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:19.398451 2651 log.cc:1199] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/ba8f9c5ea37542f3b3f99a0c1d64b90d
I20250625 20:01:19.398523 2656 ts_tablet_manager.cc:1428] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:19.398540 2653 ts_tablet_manager.cc:1428] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:19.398658 2651 ts_tablet_manager.cc:1939] T ba8f9c5ea37542f3b3f99a0c1d64b90d P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:19.399129 2651 tablet_replica.cc:331] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:19.399227 2651 raft_consensus.cc:2241] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:19.399250 2653 tablet_bootstrap.cc:492] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:19.399293 2656 tablet_bootstrap.cc:492] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:19.399327 2651 raft_consensus.cc:2270] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:19.399608 2651 ts_tablet_manager.cc:1905] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:19.400242 2653 tablet_bootstrap.cc:654] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:19.400458 2656 tablet_bootstrap.cc:654] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:19.401289 2651 ts_tablet_manager.cc:1918] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:19.401345 2651 log.cc:1199] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/08697eef3f7f4c45ba3aacf36628e652
I20250625 20:01:19.401539 2651 ts_tablet_manager.cc:1939] T 08697eef3f7f4c45ba3aacf36628e652 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:19.401634 2655 ts_tablet_manager.cc:1428] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250625 20:01:19.401823 2655 tablet_bootstrap.cc:492] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:19.402812 2655 tablet_bootstrap.cc:654] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:19.403954 2653 tablet_bootstrap.cc:492] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:19.404042 2653 ts_tablet_manager.cc:1397] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 20:01:19.404044 2656 tablet_bootstrap.cc:492] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:19.404121 2656 ts_tablet_manager.cc:1397] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 20:01:19.404163 2655 tablet_bootstrap.cc:492] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:19.404227 2655 ts_tablet_manager.cc:1397] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250625 20:01:19.404500 2653 raft_consensus.cc:357] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.404520 2656 raft_consensus.cc:357] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.404596 2653 raft_consensus.cc:383] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:19.404606 2656 raft_consensus.cc:383] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:19.404628 2653 raft_consensus.cc:738] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:19.404639 2656 raft_consensus.cc:738] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:19.404637 2655 raft_consensus.cc:357] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.404719 2655 raft_consensus.cc:383] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:19.404754 2653 consensus_queue.cc:260] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.404747 2655 raft_consensus.cc:738] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:19.416127 2655 consensus_queue.cc:260] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.416585 2655 ts_tablet_manager.cc:1428] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20250625 20:01:19.416785 2655 tablet_bootstrap.cc:492] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:19.417876 2655 tablet_bootstrap.cc:654] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:19.417922 2653 ts_tablet_manager.cc:1428] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250625 20:01:19.415863 2364 raft_consensus.cc:491] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:19.418300 2364 raft_consensus.cc:513] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.418659 2364 leader_election.cc:290] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:19.418913 2655 tablet_bootstrap.cc:492] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:19.418995 2655 ts_tablet_manager.cc:1397] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250625 20:01:19.419075 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b1d8f876f744c868857b56e49dda3e7" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:19.419190 1700 raft_consensus.cc:2466] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:19.419788 2653 tablet_bootstrap.cc:492] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:19.419487 1495 leader_election.cc:304] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:19.419454 2655 raft_consensus.cc:357] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.429220 2655 raft_consensus.cc:383] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:19.404747 2656 consensus_queue.cc:260] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.429280 2655 raft_consensus.cc:738] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:19.429425 2655 consensus_queue.cc:260] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.429602 2656 ts_tablet_manager.cc:1428] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.025s user 0.001s sys 0.000s
I20250625 20:01:19.429694 2655 ts_tablet_manager.cc:1428] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20250625 20:01:19.429833 2364 raft_consensus.cc:2802] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:19.429868 2364 raft_consensus.cc:491] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:19.429893 2364 raft_consensus.cc:3058] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:19.430857 2364 raft_consensus.cc:513] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.431223 2364 leader_election.cc:290] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:19.420773 2653 tablet_bootstrap.cc:654] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:19.427369 2503 raft_consensus.cc:491] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:19.419659 1423 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b1d8f876f744c868857b56e49dda3e7" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:19.435762 2503 raft_consensus.cc:513] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.435843 1423 raft_consensus.cc:2466] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:19.432674 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b1d8f876f744c868857b56e49dda3e7" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:19.432133 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b1d8f876f744c868857b56e49dda3e7" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:19.435978 1700 raft_consensus.cc:3058] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:19.436038 1424 raft_consensus.cc:3058] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:19.436146 2503 leader_election.cc:290] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:19.436954 1700 raft_consensus.cc:2466] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:19.436995 1424 raft_consensus.cc:2466] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:19.437191 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d55b31ba2d2f474ca4871ee6049f24bf" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:19.437242 1495 leader_election.cc:304] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:19.437306 1560 raft_consensus.cc:2466] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:19.437582 1634 leader_election.cc:304] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:19.437636 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d55b31ba2d2f474ca4871ee6049f24bf" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:19.437737 1424 raft_consensus.cc:2466] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:19.437475 2676 raft_consensus.cc:2802] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:19.438129 2676 raft_consensus.cc:695] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Becoming Leader. State: Replica: c73846c0665d4d53a274d053946107df, State: Running, Role: LEADER
I20250625 20:01:19.438292 2676 consensus_queue.cc:237] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.438665 2503 raft_consensus.cc:2802] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:19.438719 2503 raft_consensus.cc:491] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:19.438753 2503 raft_consensus.cc:3058] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:19.439456 2653 tablet_bootstrap.cc:492] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:19.439539 2653 ts_tablet_manager.cc:1397] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20250625 20:01:19.439621 2503 raft_consensus.cc:513] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.439970 2503 leader_election.cc:290] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:19.440037 2653 raft_consensus.cc:357] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.440115 2653 raft_consensus.cc:383] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:19.440145 2653 raft_consensus.cc:738] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:19.440279 2653 consensus_queue.cc:260] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.440551 2653 ts_tablet_manager.cc:1428] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:19.441232 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d55b31ba2d2f474ca4871ee6049f24bf" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:19.441339 1560 raft_consensus.cc:3058] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:19.441769 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d55b31ba2d2f474ca4871ee6049f24bf" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:19.441891 1424 raft_consensus.cc:3058] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:19.442147 1560 raft_consensus.cc:2466] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:19.442798 1424 raft_consensus.cc:2466] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:19.442845 1634 leader_election.cc:304] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:19.443002 2503 raft_consensus.cc:2802] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:19.443171 2503 raft_consensus.cc:695] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
I20250625 20:01:19.443288 2503 consensus_queue.cc:237] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.466058 2357 raft_consensus.cc:491] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:19.466193 2357 raft_consensus.cc:513] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.466632 2357 leader_election.cc:290] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:19.467133 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01c760524ea849bcbae35ed5edde6858" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:19.467139 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01c760524ea849bcbae35ed5edde6858" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:19.467272 1700 raft_consensus.cc:2466] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:19.467304 1560 raft_consensus.cc:2466] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:19.467639 1357 leader_election.cc:304] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:19.467859 2357 raft_consensus.cc:2802] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:19.467927 2357 raft_consensus.cc:491] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:19.467972 2357 raft_consensus.cc:3058] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:19.469072 2357 raft_consensus.cc:513] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.469488 2357 leader_election.cc:290] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:19.469866 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01c760524ea849bcbae35ed5edde6858" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:19.469936 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01c760524ea849bcbae35ed5edde6858" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:19.469970 1700 raft_consensus.cc:3058] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:19.470050 1560 raft_consensus.cc:3058] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:19.470865 1700 raft_consensus.cc:2466] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:19.470984 1560 raft_consensus.cc:2466] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:19.471329 1357 leader_election.cc:304] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:19.471521 2357 raft_consensus.cc:2802] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:19.471736 2357 raft_consensus.cc:695] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:19.471887 2357 consensus_queue.cc:237] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.506270 2676 raft_consensus.cc:491] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:19.506412 2676 raft_consensus.cc:513] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.506841 2676 leader_election.cc:290] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:19.507306 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb717c706f3d428bbd5bf6c8e4f839aa" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:19.507336 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb717c706f3d428bbd5bf6c8e4f839aa" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:19.507447 1700 raft_consensus.cc:2466] T eb717c706f3d428bbd5bf6c8e4f839aa P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:19.507495 1424 raft_consensus.cc:2466] T eb717c706f3d428bbd5bf6c8e4f839aa P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:19.507792 1495 leader_election.cc:304] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:19.507984 2676 raft_consensus.cc:2802] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:19.508054 2676 raft_consensus.cc:491] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:19.508093 2676 raft_consensus.cc:3058] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:19.508917 2676 raft_consensus.cc:513] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.509325 2676 leader_election.cc:290] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:19.509660 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb717c706f3d428bbd5bf6c8e4f839aa" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:19.509764 1424 raft_consensus.cc:3058] T eb717c706f3d428bbd5bf6c8e4f839aa P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:19.509897 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb717c706f3d428bbd5bf6c8e4f839aa" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:19.509989 1701 raft_consensus.cc:3058] T eb717c706f3d428bbd5bf6c8e4f839aa P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:19.510673 1424 raft_consensus.cc:2466] T eb717c706f3d428bbd5bf6c8e4f839aa P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:19.510830 1701 raft_consensus.cc:2466] T eb717c706f3d428bbd5bf6c8e4f839aa P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:19.510999 1497 leader_election.cc:304] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:19.511173 2676 raft_consensus.cc:2802] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:19.511265 2676 raft_consensus.cc:695] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Becoming Leader. State: Replica: c73846c0665d4d53a274d053946107df, State: Running, Role: LEADER
I20250625 20:01:19.511394 2676 consensus_queue.cc:237] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.544019 2676 raft_consensus.cc:491] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:19.544380 2676 raft_consensus.cc:513] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.544775 2676 leader_election.cc:290] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:19.545116 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e44908760f6245a282f72cec3b5773fe" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:19.545264 1424 raft_consensus.cc:2466] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:19.545258 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e44908760f6245a282f72cec3b5773fe" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:19.545406 1701 raft_consensus.cc:2466] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:19.545574 1497 leader_election.cc:304] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:19.545778 2676 raft_consensus.cc:2802] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:19.545833 2676 raft_consensus.cc:491] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:19.545908 2676 raft_consensus.cc:3058] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:19.546679 2676 raft_consensus.cc:513] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:19.546981 2676 leader_election.cc:290] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:19.547268 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e44908760f6245a282f72cec3b5773fe" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:19.547371 1424 raft_consensus.cc:3058] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:19.547351 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e44908760f6245a282f72cec3b5773fe" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:19.547452 1701 raft_consensus.cc:3058] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:19.548182 1424 raft_consensus.cc:2466] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:19.548427 1701 raft_consensus.cc:2466] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:19.548437 1497 leader_election.cc:304] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:19.548588 2676 raft_consensus.cc:2802] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:19.548753 2676 raft_consensus.cc:695] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Becoming Leader. State: Replica: c73846c0665d4d53a274d053946107df, State: Running, Role: LEADER
I20250625 20:01:19.548915 2676 consensus_queue.cc:237] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
W20250625 20:01:19.556118 1495 connection.cc:537] server connection from 127.0.0.1:52290 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:19.556121 1634 connection.cc:537] server connection from 127.0.0.1:43822 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:19.556178 1637 connection.cc:537] client connection to 127.0.204.60:46191 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:19.556572 1357 connection.cc:537] server connection from 127.0.0.1:54590 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:19.556592 1360 connection.cc:537] client connection to 127.0.204.60:46191 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:19.557760 1497 connection.cc:537] client connection to 127.0.204.60:46191 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:19.557945 2622 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 20:01:19.557181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.204.60:46191 (local address 127.0.204.3:41359)
0625 20:01:19.557418 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0625 20:01:19.557432 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0625 20:01:19.557517 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 20:01:19.557838 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":132}
W20250625 20:01:19.558610 2621 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 20:01:19.557187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.204.60:46191 (local address 127.0.204.1:50609)
0625 20:01:19.557389 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0625 20:01:19.557403 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0625 20:01:19.557499 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 20:01:19.558511 (+ 1012us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":117}
W20250625 20:01:19.558622 1757 connection.cc:537] client connection to 127.0.204.60:46191 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:19.558990 1757 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.204.60:46191)
W20250625 20:01:19.559303 1757 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.204.60:46191)
W20250625 20:01:19.559755 1757 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.204.60:46191)
W20250625 20:01:19.559836 1757 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.204.60:46191)
I20250625 20:01:19.832931 2503 consensus_queue.cc:1035] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:19.835784 2676 consensus_queue.cc:1035] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:19.838631 2503 consensus_queue.cc:1035] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:19.841813 2364 consensus_queue.cc:1035] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:19.850914 2701 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fcdee0c0aac4013a5e3c77304b474d9)
I20250625 20:01:19.851027 2701 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:19.851423 2701 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:19.852480 2532 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:19.852569 2534 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:19.852653 2534 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:19.852782 2701 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250625 20:01:19.948314 2364 consensus_queue.cc:1035] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:19.952334 2706 consensus_queue.cc:1035] T eb717c706f3d428bbd5bf6c8e4f839aa P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:19.955052 2706 consensus_queue.cc:1035] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:19.957994 2706 consensus_queue.cc:1035] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:19.989952 2685 consensus_queue.cc:1035] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:19.993400 2685 consensus_queue.cc:1035] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:20.446229 2714 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1fcdee0c0aac4013a5e3c77304b474d9)
I20250625 20:01:20.446357 2714 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:20.446744 2714 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:20.447881 2534 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:01:20.447997 2532 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:20.448074 2532 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:20.448215 2714 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250625 20:01:20.661350 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.60:46191
--webserver_interface=127.0.204.60
--webserver_port=43997
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 20:01:20.770990 2721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:01:20.771282 2721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:20.771350 2721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:01:20.774809 2721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:01:20.774878 2721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:20.774899 2721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:01:20.774915 2721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:01:20.778769 2721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.60:46191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.204.60
--webserver_port=43997
--never_fsync=true
--heap_profile_path=/tmp/kudu.2721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:20.779879 2721 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:20.780979 2721 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:01:20.787919 2729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:20.787922 2728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:20.788152 2721 server_base.cc:1048] running on GCE node
W20250625 20:01:20.788154 2731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:20.788846 2721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:20.789783 2721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:20.790973 2721 hybrid_clock.cc:648] HybridClock initialized: now 1750881680790949 us; error 37 us; skew 500 ppm
I20250625 20:01:20.793002 2721 webserver.cc:469] Webserver started at http://127.0.204.60:43997/ using document root <none> and password file <none>
I20250625 20:01:20.793572 2721 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:20.793629 2721 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:20.797160 2721 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20250625 20:01:20.799245 2737 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:20.800374 2721 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250625 20:01:20.800491 2721 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
uuid: "1fcdee0c0aac4013a5e3c77304b474d9"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
I20250625 20:01:20.800882 2721 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:20.823812 2721 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:20.824471 2721 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:20.824631 2721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:20.831681 2721 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.60:46191
I20250625 20:01:20.831691 2789 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.60:46191 every 8 connection(s)
I20250625 20:01:20.832835 2721 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/info.pb
I20250625 20:01:20.837230 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 2721
I20250625 20:01:20.837584 2790 sys_catalog.cc:263] Verifying existing consensus state
I20250625 20:01:20.839627 2790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Bootstrap starting.
I20250625 20:01:20.851084 2790 log.cc:826] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:20.898726 2790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=100 overwritten=0 applied=100 ignored=0} inserts{seen=67 ignored=0} mutations{seen=88 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:01:20.899083 2790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Bootstrap complete.
I20250625 20:01:20.901679 2790 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:20.902009 2790 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcdee0c0aac4013a5e3c77304b474d9, State: Initialized, Role: FOLLOWER
I20250625 20:01:20.902428 2790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 100, Last appended: 6.100, Last appended by leader: 100, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:20.903064 2795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:20.903175 2795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:20.903434 2790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:01:20.909569 2809 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:20.909631 2809 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: loading cluster ID for follower catalog manager: success
I20250625 20:01:20.908506 2721 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:01:20.910101 2721 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:20.912571 2809 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: acquiring CA information for follower catalog manager: success
I20250625 20:01:20.914847 2809 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 20:01:21.045092 1470 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:21.047086 2754 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" instance_seqno: 1750881665574847) as {username='slave'} at 127.0.204.1:60715; Asking this server to re-register.
I20250625 20:01:21.047669 1470 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:21.049150 2754 ts_manager.cc:194] Registered new tserver with Master: 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:21.056039 1607 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:21.057299 2754 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c73846c0665d4d53a274d053946107df" instance_seqno: 1750881665791183) as {username='slave'} at 127.0.204.2:47923; Asking this server to re-register.
I20250625 20:01:21.057699 1607 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:21.058763 2754 ts_manager.cc:194] Registered new tserver with Master: c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:21.074620 2712 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:21.074769 2712 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.075232 2712 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:01:21.075968 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:01:21.076087 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:01:21.076148 1637 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:01:21.076297 2712 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:01:21.275239 1747 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:21.276376 2754 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" instance_seqno: 1750881665979916) as {username='slave'} at 127.0.204.3:35601; Asking this server to re-register.
I20250625 20:01:21.276803 1747 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:21.277899 2754 ts_manager.cc:194] Registered new tserver with Master: 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:21.325733 2795 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:21.325947 2795 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:21.327268 2795 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
W20250625 20:01:21.327796 2739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:21.328581 2739 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:21.330941 2557 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 100 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" is_pre_election: true
I20250625 20:01:21.331082 2557 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fcdee0c0aac4013a5e3c77304b474d9 in term 6.
I20250625 20:01:21.331528 2738 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0; no voters: 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:21.331813 2795 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250625 20:01:21.331892 2795 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:21.331923 2795 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 6 FOLLOWER]: Advancing to term 7
I20250625 20:01:21.333598 2795 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:21.333937 2795 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 7 election: Requested vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
I20250625 20:01:21.334329 2557 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 100 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0"
I20250625 20:01:21.334460 2557 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 6 FOLLOWER]: Advancing to term 7
W20250625 20:01:21.334838 2739 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:21.335422 2557 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcdee0c0aac4013a5e3c77304b474d9 in term 7.
I20250625 20:01:21.335685 2738 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0; no voters: 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:21.335875 2795 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 7 FOLLOWER]: Leader election won for term 7
I20250625 20:01:21.336185 2795 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 7 LEADER]: Becoming Leader. State: Replica: 1fcdee0c0aac4013a5e3c77304b474d9, State: Running, Role: LEADER
I20250625 20:01:21.336512 2795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 6.100, Last appended by leader: 100, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:21.337976 2822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fcdee0c0aac4013a5e3c77304b474d9. Latest consensus state: current_term: 7 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:21.338088 2822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:21.338368 2824 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:01:21.340086 2824 catalog_manager.cc:671] Loaded metadata for table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]
I20250625 20:01:21.340535 2824 catalog_manager.cc:671] Loaded metadata for table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]
I20250625 20:01:21.340729 2824 catalog_manager.cc:671] Loaded metadata for table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]
I20250625 20:01:21.340914 2824 catalog_manager.cc:671] Loaded metadata for table default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4]
I20250625 20:01:21.341095 2824 catalog_manager.cc:671] Loaded metadata for table default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]
I20250625 20:01:21.341280 2824 catalog_manager.cc:671] Loaded metadata for table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]
I20250625 20:01:21.341475 2824 catalog_manager.cc:671] Loaded metadata for table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996]
I20250625 20:01:21.341660 2824 catalog_manager.cc:671] Loaded metadata for table default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5]
I20250625 20:01:21.341843 2824 catalog_manager.cc:671] Loaded metadata for table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583]
I20250625 20:01:21.342047 2824 catalog_manager.cc:671] Loaded metadata for table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277]
I20250625 20:01:21.342419 2824 catalog_manager.cc:671] Loaded metadata for table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]
I20250625 20:01:21.342646 2824 catalog_manager.cc:671] Loaded metadata for table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f]
I20250625 20:01:21.342854 2824 catalog_manager.cc:671] Loaded metadata for table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]
I20250625 20:01:21.343050 2824 catalog_manager.cc:671] Loaded metadata for table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]
I20250625 20:01:21.343236 2824 catalog_manager.cc:671] Loaded metadata for table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]
I20250625 20:01:21.344516 2824 tablet_loader.cc:96] loaded metadata for tablet 01c760524ea849bcbae35ed5edde6858 (table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa])
I20250625 20:01:21.344748 2824 tablet_loader.cc:96] loaded metadata for tablet 06ef5241d21449169f228cfa37ff91b4 (table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996])
I20250625 20:01:21.345433 2824 tablet_loader.cc:96] loaded metadata for tablet 4ec346c593cc448fbeb6d257ecc8dc52 (table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35])
I20250625 20:01:21.345633 2824 tablet_loader.cc:96] loaded metadata for tablet 6c8e829c96474e94a8de9e7e32b48350 (table default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4])
I20250625 20:01:21.345901 2824 tablet_loader.cc:96] loaded metadata for tablet 78631bcca66d4b16b344e57f8a1cc291 (table default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5])
I20250625 20:01:21.346019 2824 tablet_loader.cc:96] loaded metadata for tablet 7b1d8f876f744c868857b56e49dda3e7 (table default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7])
I20250625 20:01:21.346889 2824 tablet_loader.cc:96] loaded metadata for tablet bb6c755fce6640d2a0b13a449d3adfb9 (table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583])
I20250625 20:01:21.347054 2824 tablet_loader.cc:96] loaded metadata for tablet c00e8f8653bd42fda7bef627e99930cc (table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f])
I20250625 20:01:21.347206 2824 tablet_loader.cc:96] loaded metadata for tablet c368f57ac08446e5a34b67f11ac51dc8 (table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e])
I20250625 20:01:21.347473 2824 tablet_loader.cc:96] loaded metadata for tablet cdb2a543898642d3abc3afa850178120 (table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277])
I20250625 20:01:21.347644 2824 tablet_loader.cc:96] loaded metadata for tablet d4cd08f25a694ab9949c119997cad26d (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809])
I20250625 20:01:21.347846 2824 tablet_loader.cc:96] loaded metadata for tablet d55b31ba2d2f474ca4871ee6049f24bf (table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8])
I20250625 20:01:21.348086 2824 tablet_loader.cc:96] loaded metadata for tablet d8dd4c0e80974a67b63293d83760fd8b (table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36])
I20250625 20:01:21.348387 2824 tablet_loader.cc:96] loaded metadata for tablet e44908760f6245a282f72cec3b5773fe (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0])
I20250625 20:01:21.348536 2824 tablet_loader.cc:96] loaded metadata for tablet eb717c706f3d428bbd5bf6c8e4f839aa (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1])
I20250625 20:01:21.348652 2824 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:01:21.349363 2824 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:21.349407 2824 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:01:21.350198 2824 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:01:21.350827 2824 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Loaded TSK: 0
I20250625 20:01:21.351526 2824 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:01:21.352178 2824 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 20:01:21.352782 2824 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 21
I20250625 20:01:21.413594 2754 master_service.cc:948] ReplaceTablet: received request to replace tablet d55b31ba2d2f474ca4871ee6049f24bf from {username='slave'} at 127.0.0.1:45934
I20250625 20:01:21.413709 2751 master_service.cc:948] ReplaceTablet: received request to replace tablet c368f57ac08446e5a34b67f11ac51dc8 from {username='slave'} at 127.0.0.1:45934
I20250625 20:01:21.414099 2753 master_service.cc:948] ReplaceTablet: received request to replace tablet c368f57ac08446e5a34b67f11ac51dc8 from {username='slave'} at 127.0.0.1:45934
I20250625 20:01:21.414290 2745 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45934:
name: "default.Table_c95f91afca114793990044d26cd96c64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:21.414772 2752 master_service.cc:948] ReplaceTablet: received request to replace tablet 01c760524ea849bcbae35ed5edde6858 from {username='slave'} at 127.0.0.1:45934
I20250625 20:01:21.415521 2748 master_service.cc:948] ReplaceTablet: received request to replace tablet 4ec346c593cc448fbeb6d257ecc8dc52 from {username='slave'} at 127.0.0.1:45934
I20250625 20:01:21.416044 2746 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45934:
name: "default.Table_ac867d9f2ac54f09a848a71cd7a01d49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:21.416211 2750 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45934:
name: "default.Table_a31a093a024346aab31bc8d55a00253d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:21.416453 2749 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45934:
name: "default.Table_d4ed5ace6c144df6af9ff3155b29c5bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:21.416697 2747 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45934:
name: "default.Table_3ef4196a2a204be1bb572abdbef3264c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:21.417876 2557 raft_consensus.cc:1273] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Refusing update from remote peer 1fcdee0c0aac4013a5e3c77304b474d9: Log matching property violated. Preceding OpId in replica: term: 6 index: 100. Preceding OpId from leader: term: 7 index: 102. (index mismatch)
I20250625 20:01:21.418721 2822 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
W20250625 20:01:21.419200 2739 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 -> Peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Couldn't send request to peer 5852fb53503b438c92f294e0c44c916d. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 20:01:21.421039 2826 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fcdee0c0aac4013a5e3c77304b474d9. Latest consensus state: current_term: 7 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:21.421241 2826 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
F20250625 20:01:21.421917 2808 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 6 out of 13 received events: NotificationEvent(eventId=22, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_063972c2ee4b4e468cf982177f09f91a, message={"server":"","servicePrincipal":"","db":"default","table":"table_063972c2ee4b4e468cf982177f09f91a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_063972c2ee4b4e468cf982177f09f91a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_063972c2ee4b4e468cf982177f09f91a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_063972c2ee4b4e468cf982177f09f91a\",\"kudu.table_id\":\"a054c42074394f1db21dfcf2c55d99b5\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_01c13560571f419e9b6f45edee3a2a37, message={"server":"","servicePrincipal":"","db":"default","table":"table_01c13560571f419e9b6f45edee3a2a37","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_01c13560571f419e9b6f45edee3a2a37\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_01c13560571f419e9b6f45edee3a2a37\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_01c13560571f419e9b6f45edee3a2a37\",\"kudu.table_id\":\"aee290f61c8f4d7b93d4cd7192eb9277\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_a31a093a024346aab31bc8d55a00253d, message={"server":"","servicePrincipal":"","db":"default","table":"table_a31a093a024346aab31bc8d55a00253d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a31a093a024346aab31bc8d55a00253d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_a31a093a024346aab31bc8d55a00253d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a31a093a024346aab31bc8d55a00253d\",\"kudu.table_id\":\"81aee385d1404c82aaeebee220cb9e68\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_60554f72b1aa48aa93aaea511a985fe7, message={"server":"","servicePrincipal":"","db":"default","table":"table_60554f72b1aa48aa93aaea511a985fe7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_60554f72b1aa48aa93aaea511a985fe7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881667},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_60554f72b1aa48aa93aaea511a985fe7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_60554f72b1aa48aa93aaea511a985fe7\",\"kudu.table_id\":\"e46f78618ca648e19c3935d0bf874a35\",\"transient_lastDdlTime\":\"1750881667\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250625 20:01:21.422807 2795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:21.422904 2795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:21.423440 2826 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:21.423552 2826 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:21.423794 2825 mvcc.cc:204] Tried to move back new op lower bound from 7171611367090360320 to 7171611366756626432. Current Snapshot: MvccSnapshot[applied={T|T < 7171611367083069440}]
@ 0x7f7ca050adcd google::LogMessage::Fail() at ??:0
I20250625 20:01:21.425505 2751 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Sending DeleteTablet for 3 replicas of tablet c368f57ac08446e5a34b67f11ac51dc8
I20250625 20:01:21.425712 2748 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Sending DeleteTablet for 3 replicas of tablet 4ec346c593cc448fbeb6d257ecc8dc52
I20250625 20:01:21.425941 2754 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Sending DeleteTablet for 3 replicas of tablet d55b31ba2d2f474ca4871ee6049f24bf
I20250625 20:01:21.426131 2752 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Sending DeleteTablet for 3 replicas of tablet 01c760524ea849bcbae35ed5edde6858
I20250625 20:01:21.426338 2753 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45934:
name: "default.Table_bb2207463de346c3b1a4271f21eed795"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:21.426540 2751 catalog_manager.cc:6652] ReplaceTablet: tablet c368f57ac08446e5a34b67f11ac51dc8 deleted and replaced by tablet 01f62ece8947463daef2b44571d4dec0
I20250625 20:01:21.426756 2751 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45934:
name: "default.Table_117db1e3446d41099601e6b995a36d74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f7ca050eb93 google::LogMessage::SendToLog() at ??:0
I20250625 20:01:21.428639 2752 catalog_manager.cc:6652] ReplaceTablet: tablet 01c760524ea849bcbae35ed5edde6858 deleted and replaced by tablet 6b660df5bca642af848d7464859ac7fc
I20250625 20:01:21.428783 2748 catalog_manager.cc:6652] ReplaceTablet: tablet 4ec346c593cc448fbeb6d257ecc8dc52 deleted and replaced by tablet d1b4020c408349719f7533e4b6c0d1cd
I20250625 20:01:21.428826 2752 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45934:
name: "default.Table_e91ea2924c1b40fc9d7610784c293467"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:21.428930 2748 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45934:
name: "default.Table_ea337ae9a0e74819be0ba8940e1828e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:21.426548 2754 catalog_manager.cc:6652] ReplaceTablet: tablet d55b31ba2d2f474ca4871ee6049f24bf deleted and replaced by tablet 3dd7cedb1c3542c086ebae940840147b
I20250625 20:01:21.429672 2754 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45934:
table { table_name: "default.Table_1d0eb88cbdfd4b16bb66bfb3d502672b" } new_table_name: "default.Table_dd960e4b8a9c42309150252433773c76" modify_external_catalogs: true
@ 0x7f7ca050a7cc google::LogMessage::Flush() at ??:0
@ 0x7f7ca050aa11 google::LogMessage::~LogMessage() at ??:0
I20250625 20:01:21.435736 1401 tablet_service.cc:1515] Processing DeleteTablet for tablet 01c760524ea849bcbae35ed5edde6858 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b660df5bca642af848d7464859ac7fc) from {username='slave'} at 127.0.0.1:54592
I20250625 20:01:21.435801 1675 tablet_service.cc:1515] Processing DeleteTablet for tablet c368f57ac08446e5a34b67f11ac51dc8 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f62ece8947463daef2b44571d4dec0) from {username='slave'} at 127.0.0.1:43828
I20250625 20:01:21.435843 1672 tablet_service.cc:1515] Processing DeleteTablet for tablet d55b31ba2d2f474ca4871ee6049f24bf with delete_type TABLET_DATA_DELETED (replaced by tablet 3dd7cedb1c3542c086ebae940840147b) from {username='slave'} at 127.0.0.1:43828
I20250625 20:01:21.435981 1397 tablet_service.cc:1515] Processing DeleteTablet for tablet c368f57ac08446e5a34b67f11ac51dc8 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f62ece8947463daef2b44571d4dec0) from {username='slave'} at 127.0.0.1:54592
I20250625 20:01:21.436048 1673 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ec346c593cc448fbeb6d257ecc8dc52 with delete_type TABLET_DATA_DELETED (replaced by tablet d1b4020c408349719f7533e4b6c0d1cd) from {username='slave'} at 127.0.0.1:43828
I20250625 20:01:21.436092 1399 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ec346c593cc448fbeb6d257ecc8dc52 with delete_type TABLET_DATA_DELETED (replaced by tablet d1b4020c408349719f7533e4b6c0d1cd) from {username='slave'} at 127.0.0.1:54592
I20250625 20:01:21.436167 1398 tablet_service.cc:1515] Processing DeleteTablet for tablet d55b31ba2d2f474ca4871ee6049f24bf with delete_type TABLET_DATA_DELETED (replaced by tablet 3dd7cedb1c3542c086ebae940840147b) from {username='slave'} at 127.0.0.1:54592
I20250625 20:01:21.436159 1676 tablet_service.cc:1515] Processing DeleteTablet for tablet 01c760524ea849bcbae35ed5edde6858 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b660df5bca642af848d7464859ac7fc) from {username='slave'} at 127.0.0.1:43828
I20250625 20:01:21.436647 1676 tablet_service.cc:1468] Processing CreateTablet for tablet 3dd7cedb1c3542c086ebae940840147b (DEFAULT_TABLE table=default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:21.436909 1676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dd7cedb1c3542c086ebae940840147b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:21.438612 1537 tablet_service.cc:1515] Processing DeleteTablet for tablet d55b31ba2d2f474ca4871ee6049f24bf with delete_type TABLET_DATA_DELETED (replaced by tablet 3dd7cedb1c3542c086ebae940840147b) from {username='slave'} at 127.0.0.1:52298
I20250625 20:01:21.438632 1401 tablet_service.cc:1468] Processing CreateTablet for tablet d1b4020c408349719f7533e4b6c0d1cd (DEFAULT_TABLE table=default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:21.438874 1401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1b4020c408349719f7533e4b6c0d1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:21.439008 1539 tablet_service.cc:1515] Processing DeleteTablet for tablet c368f57ac08446e5a34b67f11ac51dc8 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f62ece8947463daef2b44571d4dec0) from {username='slave'} at 127.0.0.1:52298
I20250625 20:01:21.439128 1534 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ec346c593cc448fbeb6d257ecc8dc52 with delete_type TABLET_DATA_DELETED (replaced by tablet d1b4020c408349719f7533e4b6c0d1cd) from {username='slave'} at 127.0.0.1:52298
I20250625 20:01:21.439147 1537 tablet_service.cc:1468] Processing CreateTablet for tablet 6b660df5bca642af848d7464859ac7fc (DEFAULT_TABLE table=default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:21.439327 1398 tablet_service.cc:1468] Processing CreateTablet for tablet 3dd7cedb1c3542c086ebae940840147b (DEFAULT_TABLE table=default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:21.439366 1537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b660df5bca642af848d7464859ac7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:21.439353 1538 tablet_service.cc:1468] Processing CreateTablet for tablet 3dd7cedb1c3542c086ebae940840147b (DEFAULT_TABLE table=default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:21.439535 1398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dd7cedb1c3542c086ebae940840147b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:21.439558 1538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dd7cedb1c3542c086ebae940840147b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:21.440937 1536 tablet_service.cc:1515] Processing DeleteTablet for tablet 01c760524ea849bcbae35ed5edde6858 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b660df5bca642af848d7464859ac7fc) from {username='slave'} at 127.0.0.1:52298
I20250625 20:01:21.441568 1399 tablet_service.cc:1468] Processing CreateTablet for tablet 6b660df5bca642af848d7464859ac7fc (DEFAULT_TABLE table=default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:21.441623 1532 tablet_service.cc:1468] Processing CreateTablet for tablet 01f62ece8947463daef2b44571d4dec0 (DEFAULT_TABLE table=default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:21.441761 1399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b660df5bca642af848d7464859ac7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:21.441860 1533 tablet_service.cc:1468] Processing CreateTablet for tablet d1b4020c408349719f7533e4b6c0d1cd (DEFAULT_TABLE table=default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:21.441926 1532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01f62ece8947463daef2b44571d4dec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:21.442037 1533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1b4020c408349719f7533e4b6c0d1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:21.443783 1397 tablet_service.cc:1468] Processing CreateTablet for tablet 01f62ece8947463daef2b44571d4dec0 (DEFAULT_TABLE table=default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:21.443984 1397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01f62ece8947463daef2b44571d4dec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:21.444200 2844 tablet_replica.cc:331] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:21.444352 2846 tablet_bootstrap.cc:492] T 3dd7cedb1c3542c086ebae940840147b P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:21.444372 2844 raft_consensus.cc:2241] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:21.444476 2844 raft_consensus.cc:2270] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:21.444828 2844 ts_tablet_manager.cc:1905] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f7ca54a3c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250625 20:01:21.445437 2846 tablet_bootstrap.cc:654] T 3dd7cedb1c3542c086ebae940840147b P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:21.445971 2845 tablet_replica.cc:331] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:21.446084 2845 raft_consensus.cc:2241] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:21.446182 2845 raft_consensus.cc:2270] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:21.446331 1673 tablet_service.cc:1468] Processing CreateTablet for tablet 01f62ece8947463daef2b44571d4dec0 (DEFAULT_TABLE table=default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:21.446523 2845 ts_tablet_manager.cc:1905] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:21.446560 1673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01f62ece8947463daef2b44571d4dec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:21.446990 2844 ts_tablet_manager.cc:1918] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:21.447046 2844 log.cc:1199] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/c368f57ac08446e5a34b67f11ac51dc8
I20250625 20:01:21.447293 2844 ts_tablet_manager.cc:1939] T c368f57ac08446e5a34b67f11ac51dc8 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:21.447901 2844 tablet_replica.cc:331] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:21.448006 2844 raft_consensus.cc:2241] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:21.448249 2844 raft_consensus.cc:2270] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:21.448513 1675 tablet_service.cc:1468] Processing CreateTablet for tablet d1b4020c408349719f7533e4b6c0d1cd (DEFAULT_TABLE table=default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:21.448580 2844 ts_tablet_manager.cc:1905] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:21.448710 2847 tablet_bootstrap.cc:492] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:21.448751 1675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1b4020c408349719f7533e4b6c0d1cd. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f7ca54a2ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250625 20:01:21.449657 2847 tablet_bootstrap.cc:654] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:21.450263 2843 tablet_replica.cc:331] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:21.450273 2844 ts_tablet_manager.cc:1918] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:21.450327 2844 log.cc:1199] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/d55b31ba2d2f474ca4871ee6049f24bf
I20250625 20:01:21.450378 2843 raft_consensus.cc:2241] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:21.450543 2844 ts_tablet_manager.cc:1939] T d55b31ba2d2f474ca4871ee6049f24bf P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:21.450563 2843 raft_consensus.cc:2270] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:21.450876 2843 ts_tablet_manager.cc:1905] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:21.450996 2844 tablet_replica.cc:331] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:21.451105 2844 raft_consensus.cc:2241] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:21.451189 2844 raft_consensus.cc:2270] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:21.451478 2844 ts_tablet_manager.cc:1905] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f7ca54a230b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250625 20:01:21.452477 2739 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet c368f57ac08446e5a34b67f11ac51dc8 (table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]) successfully deleted
I20250625 20:01:21.452740 2739 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet d55b31ba2d2f474ca4871ee6049f24bf (table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]) successfully deleted
I20250625 20:01:21.452750 2843 ts_tablet_manager.cc:1918] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:21.452806 2843 log.cc:1199] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/01c760524ea849bcbae35ed5edde6858
I20250625 20:01:21.453027 2843 ts_tablet_manager.cc:1939] T 01c760524ea849bcbae35ed5edde6858 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:21.452997 1676 tablet_service.cc:1468] Processing CreateTablet for tablet 6b660df5bca642af848d7464859ac7fc (DEFAULT_TABLE table=default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:21.453063 2844 ts_tablet_manager.cc:1918] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:21.453112 2844 log.cc:1199] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/4ec346c593cc448fbeb6d257ecc8dc52
I20250625 20:01:21.453203 1676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b660df5bca642af848d7464859ac7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:21.453320 2844 ts_tablet_manager.cc:1939] T 4ec346c593cc448fbeb6d257ecc8dc52 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:21.453540 2843 tablet_replica.cc:331] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:21.453639 2843 raft_consensus.cc:2241] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:21.453644 2741 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet 01c760524ea849bcbae35ed5edde6858 (table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]) successfully deleted
I20250625 20:01:21.453771 2844 tablet_replica.cc:331] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:21.453786 2843 raft_consensus.cc:2270] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:21.453866 2844 raft_consensus.cc:2241] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:21.453910 2739 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet 4ec346c593cc448fbeb6d257ecc8dc52 (table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]) successfully deleted
I20250625 20:01:21.453940 2844 raft_consensus.cc:2270] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:21.454087 2843 ts_tablet_manager.cc:1905] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:21.454279 2844 ts_tablet_manager.cc:1905] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f7ca54a7a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56112da08752 std::function<>::operator()() at ??:0
I20250625 20:01:21.455807 2845 ts_tablet_manager.cc:1918] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:21.455835 2844 ts_tablet_manager.cc:1918] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:21.455880 2844 log.cc:1199] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/01c760524ea849bcbae35ed5edde6858
I20250625 20:01:21.455880 2845 log.cc:1199] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/d55b31ba2d2f474ca4871ee6049f24bf
I20250625 20:01:21.456063 2848 tablet_bootstrap.cc:492] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:21.456090 2844 ts_tablet_manager.cc:1939] T 01c760524ea849bcbae35ed5edde6858 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:21.456096 2845 ts_tablet_manager.cc:1939] T d55b31ba2d2f474ca4871ee6049f24bf P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:21.456580 2843 ts_tablet_manager.cc:1918] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:21.456645 2843 log.cc:1199] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/c368f57ac08446e5a34b67f11ac51dc8
I20250625 20:01:21.456693 2739 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet 01c760524ea849bcbae35ed5edde6858 (table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]) successfully deleted
I20250625 20:01:21.456779 2845 tablet_replica.cc:331] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:21.456691 2738 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet d55b31ba2d2f474ca4871ee6049f24bf (table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]) successfully deleted
I20250625 20:01:21.456856 2843 ts_tablet_manager.cc:1939] T c368f57ac08446e5a34b67f11ac51dc8 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:21.456880 2845 raft_consensus.cc:2241] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:21.456969 2845 raft_consensus.cc:2270] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:21.457276 2845 ts_tablet_manager.cc:1905] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:21.457324 2843 tablet_replica.cc:331] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:21.457413 2843 raft_consensus.cc:2241] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:21.457417 2741 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet c368f57ac08446e5a34b67f11ac51dc8 (table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]) successfully deleted
I20250625 20:01:21.457574 2843 raft_consensus.cc:2270] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:21.457854 2843 ts_tablet_manager.cc:1905] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f7ca195c0a1 kudu::Thread::SuperviseThread() at ??:0
I20250625 20:01:21.457978 2848 tablet_bootstrap.cc:654] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:21.458057 2846 tablet_bootstrap.cc:492] T 3dd7cedb1c3542c086ebae940840147b P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:21.458127 2846 ts_tablet_manager.cc:1397] T 3dd7cedb1c3542c086ebae940840147b P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20250625 20:01:21.458562 2846 raft_consensus.cc:357] T 3dd7cedb1c3542c086ebae940840147b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.458660 2846 raft_consensus.cc:383] T 3dd7cedb1c3542c086ebae940840147b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:21.458693 2846 raft_consensus.cc:738] T 3dd7cedb1c3542c086ebae940840147b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:21.458812 2846 consensus_queue.cc:260] T 3dd7cedb1c3542c086ebae940840147b P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.458968 2847 tablet_bootstrap.cc:492] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:21.459036 2847 ts_tablet_manager.cc:1397] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250625 20:01:21.459488 2843 ts_tablet_manager.cc:1918] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:21.459468 2847 raft_consensus.cc:357] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.459546 2843 log.cc:1199] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/4ec346c593cc448fbeb6d257ecc8dc52
I20250625 20:01:21.459570 2847 raft_consensus.cc:383] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:21.459601 2847 raft_consensus.cc:738] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
@ 0x7f7ca214b6db start_thread at ??:0
I20250625 20:01:21.459961 2843 ts_tablet_manager.cc:1939] T 4ec346c593cc448fbeb6d257ecc8dc52 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:21.460419 2843 tablet_replica.cc:331] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:21.460518 2843 raft_consensus.cc:2241] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:21.460564 2741 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet 4ec346c593cc448fbeb6d257ecc8dc52 (table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]) successfully deleted
I20250625 20:01:21.460597 2843 raft_consensus.cc:2270] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:21.460881 2843 ts_tablet_manager.cc:1905] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f7c9f6c171f clone at ??:0
I20250625 20:01:21.462085 2846 ts_tablet_manager.cc:1428] T 3dd7cedb1c3542c086ebae940840147b P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 20:01:21.462126 2847 consensus_queue.cc:260] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.462765 2843 ts_tablet_manager.cc:1918] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:21.462822 2843 log.cc:1199] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/d55b31ba2d2f474ca4871ee6049f24bf
I20250625 20:01:21.463027 2843 ts_tablet_manager.cc:1939] T d55b31ba2d2f474ca4871ee6049f24bf P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:21.463574 2846 tablet_bootstrap.cc:492] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:21.463871 2845 ts_tablet_manager.cc:1918] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:21.463928 2845 log.cc:1199] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/c368f57ac08446e5a34b67f11ac51dc8
I20250625 20:01:21.464143 2845 ts_tablet_manager.cc:1939] T c368f57ac08446e5a34b67f11ac51dc8 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:21.464507 2846 tablet_bootstrap.cc:654] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:21.464726 2845 tablet_replica.cc:331] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:21.464881 2845 raft_consensus.cc:2241] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:21.464912 2847 ts_tablet_manager.cc:1428] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250625 20:01:21.465018 2845 raft_consensus.cc:2270] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:21.465461 2846 tablet_bootstrap.cc:492] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:21.465489 2845 ts_tablet_manager.cc:1905] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:21.465529 2846 ts_tablet_manager.cc:1397] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:21.465847 2848 tablet_bootstrap.cc:492] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:21.465915 2848 ts_tablet_manager.cc:1397] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250625 20:01:21.466063 2846 raft_consensus.cc:357] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.466156 2846 raft_consensus.cc:383] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:21.466182 2846 raft_consensus.cc:738] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:21.466296 2846 consensus_queue.cc:260] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.466552 2846 ts_tablet_manager.cc:1428] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:21.466507 2848 raft_consensus.cc:357] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.466603 2848 raft_consensus.cc:383] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:21.466636 2848 raft_consensus.cc:738] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:21.466696 2846 tablet_bootstrap.cc:492] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:21.466770 2848 consensus_queue.cc:260] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.467406 2845 ts_tablet_manager.cc:1918] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:21.467458 2845 log.cc:1199] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/4ec346c593cc448fbeb6d257ecc8dc52
I20250625 20:01:21.467828 2845 ts_tablet_manager.cc:1939] T 4ec346c593cc448fbeb6d257ecc8dc52 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:21.468307 2846 tablet_bootstrap.cc:654] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:21.468371 2845 tablet_replica.cc:331] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:21.468477 2845 raft_consensus.cc:2241] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:21.468559 2845 raft_consensus.cc:2270] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:21.468840 2845 ts_tablet_manager.cc:1905] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:21.469154 2846 tablet_bootstrap.cc:492] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:21.469216 2846 ts_tablet_manager.cc:1397] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250625 20:01:21.469588 2846 raft_consensus.cc:357] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.469666 2846 raft_consensus.cc:383] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:21.469692 2846 raft_consensus.cc:738] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:21.469769 2846 consensus_queue.cc:260] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.469972 2846 ts_tablet_manager.cc:1428] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:21.470093 2846 tablet_bootstrap.cc:492] T 6b660df5bca642af848d7464859ac7fc P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:21.470436 2845 ts_tablet_manager.cc:1918] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:21.470492 2845 log.cc:1199] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/01c760524ea849bcbae35ed5edde6858
I20250625 20:01:21.470701 2845 ts_tablet_manager.cc:1939] T 01c760524ea849bcbae35ed5edde6858 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:21.470706 2848 ts_tablet_manager.cc:1428] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250625 20:01:21.471043 2846 tablet_bootstrap.cc:654] T 6b660df5bca642af848d7464859ac7fc P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:21.473275 2848 tablet_bootstrap.cc:492] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:21.473564 2847 tablet_bootstrap.cc:492] T 3dd7cedb1c3542c086ebae940840147b P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:21.474660 2847 tablet_bootstrap.cc:654] T 3dd7cedb1c3542c086ebae940840147b P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:21.475476 2848 tablet_bootstrap.cc:654] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:21.475672 2846 tablet_bootstrap.cc:492] T 6b660df5bca642af848d7464859ac7fc P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:21.475793 2846 ts_tablet_manager.cc:1397] T 6b660df5bca642af848d7464859ac7fc P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20250625 20:01:21.476100 2847 tablet_bootstrap.cc:492] T 3dd7cedb1c3542c086ebae940840147b P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:21.476167 2847 ts_tablet_manager.cc:1397] T 3dd7cedb1c3542c086ebae940840147b P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 20:01:21.476213 2846 raft_consensus.cc:357] T 6b660df5bca642af848d7464859ac7fc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.476310 2846 raft_consensus.cc:383] T 6b660df5bca642af848d7464859ac7fc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:21.476334 2846 raft_consensus.cc:738] T 6b660df5bca642af848d7464859ac7fc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:21.476439 2846 consensus_queue.cc:260] T 6b660df5bca642af848d7464859ac7fc P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.476601 2847 raft_consensus.cc:357] T 3dd7cedb1c3542c086ebae940840147b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.476701 2847 raft_consensus.cc:383] T 3dd7cedb1c3542c086ebae940840147b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:21.476728 2847 raft_consensus.cc:738] T 3dd7cedb1c3542c086ebae940840147b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:21.476756 2846 ts_tablet_manager.cc:1428] T 6b660df5bca642af848d7464859ac7fc P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 20:01:21.476847 2847 consensus_queue.cc:260] T 3dd7cedb1c3542c086ebae940840147b P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.477104 2847 ts_tablet_manager.cc:1428] T 3dd7cedb1c3542c086ebae940840147b P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 20:01:21.477252 2847 tablet_bootstrap.cc:492] T 6b660df5bca642af848d7464859ac7fc P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:21.478072 2847 tablet_bootstrap.cc:654] T 6b660df5bca642af848d7464859ac7fc P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:21.481305 2848 tablet_bootstrap.cc:492] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:21.481384 2848 ts_tablet_manager.cc:1397] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20250625 20:01:21.481845 2848 raft_consensus.cc:357] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.481946 2848 raft_consensus.cc:383] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:21.481978 2848 raft_consensus.cc:738] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:21.482146 2848 consensus_queue.cc:260] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.482314 2847 tablet_bootstrap.cc:492] T 6b660df5bca642af848d7464859ac7fc P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:21.482383 2847 ts_tablet_manager.cc:1397] T 6b660df5bca642af848d7464859ac7fc P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20250625 20:01:21.482779 2847 raft_consensus.cc:357] T 6b660df5bca642af848d7464859ac7fc P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.482870 2847 raft_consensus.cc:383] T 6b660df5bca642af848d7464859ac7fc P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:21.482898 2847 raft_consensus.cc:738] T 6b660df5bca642af848d7464859ac7fc P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:21.482995 2847 consensus_queue.cc:260] T 6b660df5bca642af848d7464859ac7fc P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.483259 2847 ts_tablet_manager.cc:1428] T 6b660df5bca642af848d7464859ac7fc P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 20:01:21.483438 2847 tablet_bootstrap.cc:492] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:21.484405 2847 tablet_bootstrap.cc:654] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:21.484815 2848 ts_tablet_manager.cc:1428] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 20:01:21.484994 2848 tablet_bootstrap.cc:492] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:21.486195 2848 tablet_bootstrap.cc:654] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:21.487221 2847 tablet_bootstrap.cc:492] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:21.487298 2847 ts_tablet_manager.cc:1397] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250625 20:01:21.487776 2847 raft_consensus.cc:357] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.487874 2847 raft_consensus.cc:383] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:21.487906 2847 raft_consensus.cc:738] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:21.488024 2847 consensus_queue.cc:260] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.488286 2847 ts_tablet_manager.cc:1428] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 20:01:21.489457 2848 tablet_bootstrap.cc:492] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:21.489544 2848 ts_tablet_manager.cc:1397] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 20:01:21.489974 2848 raft_consensus.cc:357] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.490068 2848 raft_consensus.cc:383] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:21.490100 2848 raft_consensus.cc:738] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:21.490211 2848 consensus_queue.cc:260] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.491644 2848 ts_tablet_manager.cc:1428] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:21.492399 2848 tablet_bootstrap.cc:492] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:21.493425 2848 tablet_bootstrap.cc:654] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:21.494418 2848 tablet_bootstrap.cc:492] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:21.494490 2848 ts_tablet_manager.cc:1397] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250625 20:01:21.494943 2848 raft_consensus.cc:357] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.495043 2848 raft_consensus.cc:383] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:21.495072 2848 raft_consensus.cc:738] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:21.495205 2848 consensus_queue.cc:260] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.495464 2848 ts_tablet_manager.cc:1428] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 20:01:21.514731 2364 raft_consensus.cc:491] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:21.514848 2364 raft_consensus.cc:513] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.515298 2364 leader_election.cc:290] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:21.515758 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b660df5bca642af848d7464859ac7fc" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:21.515767 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b660df5bca642af848d7464859ac7fc" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:21.515908 1424 raft_consensus.cc:2466] T 6b660df5bca642af848d7464859ac7fc P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:21.515913 1700 raft_consensus.cc:2466] T 6b660df5bca642af848d7464859ac7fc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:21.516186 1497 leader_election.cc:304] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:21.516379 2364 raft_consensus.cc:2802] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:21.516438 2364 raft_consensus.cc:491] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:21.516484 2364 raft_consensus.cc:3058] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:21.517573 2364 raft_consensus.cc:513] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.517930 2364 leader_election.cc:290] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:21.518191 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b660df5bca642af848d7464859ac7fc" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:21.518191 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b660df5bca642af848d7464859ac7fc" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:21.518291 1424 raft_consensus.cc:3058] T 6b660df5bca642af848d7464859ac7fc P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:21.518291 1700 raft_consensus.cc:3058] T 6b660df5bca642af848d7464859ac7fc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:21.519018 1700 raft_consensus.cc:2466] T 6b660df5bca642af848d7464859ac7fc P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:21.519219 1424 raft_consensus.cc:2466] T 6b660df5bca642af848d7464859ac7fc P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:21.519259 1495 leader_election.cc:304] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:21.519508 2364 raft_consensus.cc:2802] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:21.519704 2364 raft_consensus.cc:695] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Becoming Leader. State: Replica: c73846c0665d4d53a274d053946107df, State: Running, Role: LEADER
I20250625 20:01:21.519867 2364 consensus_queue.cc:237] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.530320 2364 raft_consensus.cc:491] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:21.530450 2364 raft_consensus.cc:513] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.530862 2364 leader_election.cc:290] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:21.531273 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3dd7cedb1c3542c086ebae940840147b" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:21.531251 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3dd7cedb1c3542c086ebae940840147b" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:21.531411 1424 raft_consensus.cc:2466] T 3dd7cedb1c3542c086ebae940840147b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:21.531411 1700 raft_consensus.cc:2466] T 3dd7cedb1c3542c086ebae940840147b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 0.
I20250625 20:01:21.531744 1497 leader_election.cc:304] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:21.531912 2364 raft_consensus.cc:2802] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:21.531966 2364 raft_consensus.cc:491] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:21.531994 2364 raft_consensus.cc:3058] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:21.532809 2364 raft_consensus.cc:513] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.533133 2364 leader_election.cc:290] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:21.533463 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3dd7cedb1c3542c086ebae940840147b" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:21.533460 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3dd7cedb1c3542c086ebae940840147b" candidate_uuid: "c73846c0665d4d53a274d053946107df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:21.533578 1700 raft_consensus.cc:3058] T 3dd7cedb1c3542c086ebae940840147b P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:21.533607 1424 raft_consensus.cc:3058] T 3dd7cedb1c3542c086ebae940840147b P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:21.534373 1700 raft_consensus.cc:2466] T 3dd7cedb1c3542c086ebae940840147b P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:21.534570 1424 raft_consensus.cc:2466] T 3dd7cedb1c3542c086ebae940840147b P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73846c0665d4d53a274d053946107df in term 1.
I20250625 20:01:21.534816 1495 leader_election.cc:304] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:21.534993 2364 raft_consensus.cc:2802] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:21.535085 2364 raft_consensus.cc:695] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Becoming Leader. State: Replica: c73846c0665d4d53a274d053946107df, State: Running, Role: LEADER
I20250625 20:01:21.535660 2364 consensus_queue.cc:237] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
W20250625 20:01:21.629887 1497 connection.cc:537] server connection from 127.0.0.1:52298 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:21.630090 1360 connection.cc:537] server connection from 127.0.0.1:54592 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:21.630476 1497 connection.cc:537] client connection to 127.0.204.60:46191 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:21.631785 1360 connection.cc:537] client connection to 127.0.204.60:46191 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:21.632359 1757 connection.cc:537] client connection to 127.0.204.60:46191 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:21.632841 2818 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 20:01:21.631347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.204.60:46191 (local address 127.0.204.3:48785)
0625 20:01:21.631496 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0625 20:01:21.631509 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0625 20:01:21.631594 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 20:01:21.632746 (+ 1152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":62}
W20250625 20:01:21.632905 1757 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.204.60:46191)
W20250625 20:01:21.633029 1757 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.204.60:46191)
W20250625 20:01:21.633105 1757 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.204.60:46191)
W20250625 20:01:21.633180 1757 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.204.60:46191)
W20250625 20:01:21.633265 2816 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 20:01:21.631027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.204.60:46191 (local address 127.0.204.2:51991)
0625 20:01:21.631246 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0625 20:01:21.631263 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0625 20:01:21.631340 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 20:01:21.632706 (+ 1366us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127}
I20250625 20:01:21.635351 2712 raft_consensus.cc:491] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:21.635465 2712 raft_consensus.cc:513] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.635911 2712 leader_election.cc:290] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:21.636233 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01f62ece8947463daef2b44571d4dec0" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:21.636233 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01f62ece8947463daef2b44571d4dec0" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:21.636370 1560 raft_consensus.cc:2466] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:21.636370 1424 raft_consensus.cc:2466] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:21.636623 1637 leader_election.cc:304] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:21.636828 2712 raft_consensus.cc:2802] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:21.636873 2712 raft_consensus.cc:491] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:21.636897 2712 raft_consensus.cc:3058] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:21.637825 2712 raft_consensus.cc:513] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.638157 2712 leader_election.cc:290] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:21.638448 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01f62ece8947463daef2b44571d4dec0" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:21.638463 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01f62ece8947463daef2b44571d4dec0" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:21.638566 1424 raft_consensus.cc:3058] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:21.638558 1560 raft_consensus.cc:3058] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:21.639488 1424 raft_consensus.cc:2466] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:21.639503 1560 raft_consensus.cc:2466] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:21.639822 1637 leader_election.cc:304] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:21.639986 2712 raft_consensus.cc:2802] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:21.640178 2712 raft_consensus.cc:695] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
I20250625 20:01:21.640311 2712 consensus_queue.cc:237] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.647832 2712 raft_consensus.cc:491] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:21.647944 2712 raft_consensus.cc:513] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.648275 2712 leader_election.cc:290] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:21.648627 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1b4020c408349719f7533e4b6c0d1cd" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:21.648638 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1b4020c408349719f7533e4b6c0d1cd" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:21.648751 1424 raft_consensus.cc:2466] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:21.648757 1560 raft_consensus.cc:2466] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:21.649005 1634 leader_election.cc:304] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:21.649159 2712 raft_consensus.cc:2802] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:21.649215 2712 raft_consensus.cc:491] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:21.649251 2712 raft_consensus.cc:3058] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:21.650216 2712 raft_consensus.cc:513] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.650508 2712 leader_election.cc:290] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:21.650825 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1b4020c408349719f7533e4b6c0d1cd" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:21.650825 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1b4020c408349719f7533e4b6c0d1cd" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:21.650943 1424 raft_consensus.cc:3058] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:21.650943 1560 raft_consensus.cc:3058] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:21.651880 1560 raft_consensus.cc:2466] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:21.651880 1424 raft_consensus.cc:2466] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:21.652154 1637 leader_election.cc:304] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:21.652307 2712 raft_consensus.cc:2802] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:21.652395 2712 raft_consensus.cc:695] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
I20250625 20:01:21.652520 2712 consensus_queue.cc:237] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:21.938408 2878 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1fcdee0c0aac4013a5e3c77304b474d9)
I20250625 20:01:21.938534 2878 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:21.938944 2878 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:21.940124 2532 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:21.940217 2534 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:21.940290 2534 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:21.940418 2878 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250625 20:01:22.022800 2364 consensus_queue.cc:1035] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:22.026957 2364 consensus_queue.cc:1035] T 3dd7cedb1c3542c086ebae940840147b P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:22.044322 2712 consensus_queue.cc:1035] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:22.047132 2712 consensus_queue.cc:1035] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:22.059643 2364 consensus_queue.cc:1035] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:22.063333 2364 consensus_queue.cc:1035] T 6b660df5bca642af848d7464859ac7fc P c73846c0665d4d53a274d053946107df [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:22.095240 2712 consensus_queue.cc:1035] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:22.098757 2875 consensus_queue.cc:1035] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:22.536275 2512 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:01:22.536389 2512 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:22.646270 2901 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1fcdee0c0aac4013a5e3c77304b474d9)
I20250625 20:01:22.646430 2901 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:22.646914 2901 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:22.648047 2532 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:22.648240 2534 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:22.648312 2534 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:22.648455 2901 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250625 20:01:23.675100 2916 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1fcdee0c0aac4013a5e3c77304b474d9)
I20250625 20:01:23.675233 2916 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:23.675663 2916 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:23.676219 2532 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250625 20:01:23.676836 2534 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:01:23.676920 2532 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:23.676999 2532 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:23.677125 2916 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250625 20:01:24.114099 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.62:36889
--webserver_interface=127.0.204.62
--webserver_port=37479
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 20:01:24.219780 2924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:01:24.219998 2924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:24.220052 2924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:01:24.223192 2924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:01:24.223253 2924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:24.223279 2924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:01:24.223302 2924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:01:24.226977 2924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.62:36889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.204.62
--webserver_port=37479
--never_fsync=true
--heap_profile_path=/tmp/kudu.2924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:24.228080 2924 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:24.229224 2924 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:01:24.235755 2934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:24.235826 2932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:24.235826 2931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:24.236043 2924 server_base.cc:1048] running on GCE node
I20250625 20:01:24.236564 2924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:24.237475 2924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:24.238641 2924 hybrid_clock.cc:648] HybridClock initialized: now 1750881684238614 us; error 37 us; skew 500 ppm
I20250625 20:01:24.245887 2924 webserver.cc:469] Webserver started at http://127.0.204.62:37479/ using document root <none> and password file <none>
I20250625 20:01:24.246640 2924 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:24.246742 2924 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:24.251387 2924 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20250625 20:01:24.253851 2940 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:24.255041 2924 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250625 20:01:24.255151 2924 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
uuid: "5852fb53503b438c92f294e0c44c916d"
format_stamp: "Formatted at 2025-06-25 20:01:04 on dist-test-slave-lc57"
I20250625 20:01:24.255561 2924 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:24.291632 2924 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:24.292356 2924 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:24.292521 2924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:24.299480 2924 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.62:36889
I20250625 20:01:24.299486 2992 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.62:36889 every 8 connection(s)
I20250625 20:01:24.300599 2924 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/info.pb
I20250625 20:01:24.304317 2993 sys_catalog.cc:263] Verifying existing consensus state
I20250625 20:01:24.305938 2993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Bootstrap starting.
I20250625 20:01:24.310108 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 2924
I20250625 20:01:24.317027 2993 log.cc:826] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:24.363874 2993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Bootstrap replayed 1/1 log segments. Stats: ops{read=94 overwritten=1 applied=93 ignored=0} inserts{seen=62 ignored=0} mutations{seen=78 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:01:24.364275 2993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Bootstrap complete.
I20250625 20:01:24.366927 2993 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:24.367265 2993 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5852fb53503b438c92f294e0c44c916d, State: Initialized, Role: FOLLOWER
I20250625 20:01:24.367695 2993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 93, Last appended: 5.93, Last appended by leader: 93, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:24.368187 2997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:24.368324 2997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:24.368908 2993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:01:24.375268 3011 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:24.375335 3011 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: loading cluster ID for follower catalog manager: success
I20250625 20:01:24.378172 3011 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: acquiring CA information for follower catalog manager: success
I20250625 20:01:24.380118 3011 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 20:01:24.391384 2924 master_runner.cc:186] Error getting master registration for 127.0.204.60:46191: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111),
I20250625 20:01:24.393395 2924 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:24.532367 1608 heartbeater.cc:344] Connected to a master server at 127.0.204.62:36889
I20250625 20:01:24.534250 2957 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c73846c0665d4d53a274d053946107df" instance_seqno: 1750881665791183) as {username='slave'} at 127.0.204.2:43595; Asking this server to re-register.
I20250625 20:01:24.534761 1608 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:24.535919 2957 ts_manager.cc:194] Registered new tserver with Master: c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:24.607901 1471 heartbeater.cc:344] Connected to a master server at 127.0.204.62:36889
I20250625 20:01:24.609086 2957 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" instance_seqno: 1750881665574847) as {username='slave'} at 127.0.204.1:46883; Asking this server to re-register.
I20250625 20:01:24.609501 1471 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:24.610419 2957 ts_manager.cc:194] Registered new tserver with Master: 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:24.658301 1748 heartbeater.cc:344] Connected to a master server at 127.0.204.62:36889
I20250625 20:01:24.659478 2957 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" instance_seqno: 1750881665979916) as {username='slave'} at 127.0.204.3:45993; Asking this server to re-register.
I20250625 20:01:24.659896 1748 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:24.660813 2957 ts_manager.cc:194] Registered new tserver with Master: 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:24.830844 2997 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:24.831043 2997 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:24.832254 2997 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:24.832634 2557 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 6 candidate_status { last_received { term: 5 index: 93 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" is_pre_election: true
W20250625 20:01:24.832732 2944 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:24.832806 2557 raft_consensus.cc:2366] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5852fb53503b438c92f294e0c44c916d for earlier term 6. Current term is 7.
I20250625 20:01:24.833261 2941 leader_election.cc:400] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 6 pre-election: Vote denied by peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5852fb53503b438c92f294e0c44c916d for earlier term 6. Current term is 7.
I20250625 20:01:24.833316 2941 leader_election.cc:403] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250625 20:01:24.833494 2997 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 5 FOLLOWER]: Advancing to term 7
W20250625 20:01:24.833593 2944 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:24.835337 2997 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5852fb53503b438c92f294e0c44c916d for earlier term 6. Current term is 7.
I20250625 20:01:24.905735 3023 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1fcdee0c0aac4013a5e3c77304b474d9)
I20250625 20:01:24.905889 3023 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:24.906313 3023 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:24.907609 2534 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:24.909955 2967 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" candidate_term: 8 candidate_status { last_received { term: 7 index: 106 } } ignore_live_leader: false dest_uuid: "5852fb53503b438c92f294e0c44c916d" is_pre_election: true
I20250625 20:01:24.910215 2967 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f87f9e18e5b948a4a3fb3666f106abd0 in term 7.
I20250625 20:01:24.910514 2532 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9
I20250625 20:01:24.910718 3023 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250625 20:01:24.910797 3023 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1fcdee0c0aac4013a5e3c77304b474d9)
I20250625 20:01:24.910845 3023 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 7 FOLLOWER]: Advancing to term 8
I20250625 20:01:24.911679 3023 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:24.912046 3023 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 election: Requested vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:24.912385 2967 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" candidate_term: 8 candidate_status { last_received { term: 7 index: 106 } } ignore_live_leader: false dest_uuid: "5852fb53503b438c92f294e0c44c916d"
I20250625 20:01:24.912518 2967 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 7 FOLLOWER]: Advancing to term 8
W20250625 20:01:24.912925 2534 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:24.913486 2967 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f87f9e18e5b948a4a3fb3666f106abd0 in term 8.
I20250625 20:01:24.913751 2532 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9
I20250625 20:01:24.913889 3023 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 8 FOLLOWER]: Leader election won for term 8
I20250625 20:01:24.914070 3023 raft_consensus.cc:695] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 8 LEADER]: Becoming Leader. State: Replica: f87f9e18e5b948a4a3fb3666f106abd0, State: Running, Role: LEADER
I20250625 20:01:24.914351 3023 consensus_queue.cc:237] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 7.106, Last appended by leader: 106, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:24.915262 3027 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f87f9e18e5b948a4a3fb3666f106abd0. Latest consensus state: current_term: 8 leader_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:24.915342 3027 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:24.915625 3029 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:01:24.917410 3029 catalog_manager.cc:671] Loaded metadata for table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]
I20250625 20:01:24.917821 3029 catalog_manager.cc:671] Loaded metadata for table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]
I20250625 20:01:24.917987 3029 catalog_manager.cc:671] Loaded metadata for table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]
I20250625 20:01:24.918141 3029 catalog_manager.cc:671] Loaded metadata for table default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4]
I20250625 20:01:24.918292 3029 catalog_manager.cc:671] Loaded metadata for table default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]
I20250625 20:01:24.918444 3029 catalog_manager.cc:671] Loaded metadata for table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]
I20250625 20:01:24.918592 3029 catalog_manager.cc:671] Loaded metadata for table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996]
I20250625 20:01:24.918747 3029 catalog_manager.cc:671] Loaded metadata for table default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5]
I20250625 20:01:24.918900 3029 catalog_manager.cc:671] Loaded metadata for table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583]
I20250625 20:01:24.919049 3029 catalog_manager.cc:671] Loaded metadata for table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277]
I20250625 20:01:24.919274 3029 catalog_manager.cc:671] Loaded metadata for table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]
I20250625 20:01:24.919426 3029 catalog_manager.cc:671] Loaded metadata for table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f]
I20250625 20:01:24.919572 3029 catalog_manager.cc:671] Loaded metadata for table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]
I20250625 20:01:24.919767 3029 catalog_manager.cc:671] Loaded metadata for table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]
I20250625 20:01:24.919953 3029 catalog_manager.cc:671] Loaded metadata for table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]
I20250625 20:01:24.921314 3029 tablet_loader.cc:96] loaded metadata for tablet 01f62ece8947463daef2b44571d4dec0 (table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e])
I20250625 20:01:24.921490 3029 tablet_loader.cc:96] loaded metadata for tablet 06ef5241d21449169f228cfa37ff91b4 (table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996])
I20250625 20:01:24.922086 3029 tablet_loader.cc:96] loaded metadata for tablet 3dd7cedb1c3542c086ebae940840147b (table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8])
I20250625 20:01:24.922435 3029 tablet_loader.cc:96] loaded metadata for tablet 6b660df5bca642af848d7464859ac7fc (table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa])
I20250625 20:01:24.922574 3029 tablet_loader.cc:96] loaded metadata for tablet 6c8e829c96474e94a8de9e7e32b48350 (table default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4])
I20250625 20:01:24.922863 3029 tablet_loader.cc:96] loaded metadata for tablet 78631bcca66d4b16b344e57f8a1cc291 (table default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5])
I20250625 20:01:24.922986 3029 tablet_loader.cc:96] loaded metadata for tablet 7b1d8f876f744c868857b56e49dda3e7 (table default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7])
I20250625 20:01:24.923693 3029 tablet_loader.cc:96] loaded metadata for tablet bb6c755fce6640d2a0b13a449d3adfb9 (table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583])
I20250625 20:01:24.923869 3029 tablet_loader.cc:96] loaded metadata for tablet c00e8f8653bd42fda7bef627e99930cc (table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f])
I20250625 20:01:24.924150 3029 tablet_loader.cc:96] loaded metadata for tablet cdb2a543898642d3abc3afa850178120 (table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277])
I20250625 20:01:24.924269 3029 tablet_loader.cc:96] loaded metadata for tablet d1b4020c408349719f7533e4b6c0d1cd (table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35])
I20250625 20:01:24.924386 3029 tablet_loader.cc:96] loaded metadata for tablet d4cd08f25a694ab9949c119997cad26d (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809])
I20250625 20:01:24.924656 3029 tablet_loader.cc:96] loaded metadata for tablet d8dd4c0e80974a67b63293d83760fd8b (table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36])
I20250625 20:01:24.924932 3029 tablet_loader.cc:96] loaded metadata for tablet e44908760f6245a282f72cec3b5773fe (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0])
I20250625 20:01:24.925051 3029 tablet_loader.cc:96] loaded metadata for tablet eb717c706f3d428bbd5bf6c8e4f839aa (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1])
I20250625 20:01:24.925159 3029 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:01:24.925894 3029 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:24.925935 3029 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:01:24.926764 3029 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:01:24.927434 3029 catalog_manager.cc:5966] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Loaded TSK: 0
I20250625 20:01:24.928174 3029 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:01:24.928925 3029 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 20:01:24.929561 3029 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 21
I20250625 20:01:24.973762 2547 master_service.cc:948] ReplaceTablet: received request to replace tablet d4cd08f25a694ab9949c119997cad26d from {username='slave'} at 127.0.0.1:41730
I20250625 20:01:24.973928 2546 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41730:
table { table_name: "default.Table_01c13560571f419e9b6f45edee3a2a37" } modify_external_catalogs: true
I20250625 20:01:24.974102 2546 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41730:
table { table_name: "default.Table_01c13560571f419e9b6f45edee3a2a37" } modify_external_catalogs: true
I20250625 20:01:24.974437 2540 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41730:
table { table_name: "default.Table_030bc68e40784cf2a6bd261221a0f738" } modify_external_catalogs: true
I20250625 20:01:24.974561 2540 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41730:
table { table_name: "default.Table_030bc68e40784cf2a6bd261221a0f738" } modify_external_catalogs: true
I20250625 20:01:24.974634 2542 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41730:
table { table_name: "default.Table_976af21482a74bc79eb800684cce978e" } new_table_name: "default.Table_ad0e628a3e004a9da4a42cb7d6051e3e" modify_external_catalogs: true
I20250625 20:01:24.975153 2538 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41730:
table { table_name: "default.Table_8302314b76364eadb44f02cc6d58a589" } new_table_name: "default.Table_2fcd1dbf63224757b5e1a9bad86e6a77" modify_external_catalogs: true
I20250625 20:01:24.975227 2545 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41730:
name: "default.Table_3629c083428e4ae0a977ffb16af83bc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:24.975356 2541 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41730:
table { table_name: "default.Table_ef511cc05ad8422e98275bbb9b82ecb4" } modify_external_catalogs: true
I20250625 20:01:24.975438 2541 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41730:
table { table_name: "default.Table_ef511cc05ad8422e98275bbb9b82ecb4" } modify_external_catalogs: true
I20250625 20:01:24.974561 2539 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41730:
table { table_name: "default.Table_d0f8eb84406f4d758e34b162746daa38" } new_table_name: "default.Table_a78f24eb6417409489ec4469005095b4" modify_external_catalogs: true
I20250625 20:01:24.975593 2543 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41730:
table { table_name: "default.Table_60554f72b1aa48aa93aaea511a985fe7" } modify_external_catalogs: true
I20250625 20:01:24.975656 2544 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41730:
table { table_name: "default.Table_063972c2ee4b4e468cf982177f09f91a" } new_table_name: "default.Table_980a5f7891e24ccc87c72a858cc5076f" modify_external_catalogs: true
I20250625 20:01:24.975675 2543 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41730:
table { table_name: "default.Table_60554f72b1aa48aa93aaea511a985fe7" } modify_external_catalogs: true
I20250625 20:01:24.977521 2967 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Refusing update from remote peer f87f9e18e5b948a4a3fb3666f106abd0: Log matching property violated. Preceding OpId in replica: term: 5 index: 93. Preceding OpId from leader: term: 8 index: 108. (index mismatch)
W20250625 20:01:24.978199 2534 consensus_peers.cc:487] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 -> Peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Couldn't send request to peer 1fcdee0c0aac4013a5e3c77304b474d9. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 20:01:24.978273 3027 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 93, Time since last communication: 0.000s
F20250625 20:01:24.979833 2601 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 6 out of 13 received events: NotificationEvent(eventId=22, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_063972c2ee4b4e468cf982177f09f91a, message={"server":"","servicePrincipal":"","db":"default","table":"table_063972c2ee4b4e468cf982177f09f91a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_063972c2ee4b4e468cf982177f09f91a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_063972c2ee4b4e468cf982177f09f91a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_063972c2ee4b4e468cf982177f09f91a\",\"kudu.table_id\":\"a054c42074394f1db21dfcf2c55d99b5\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_01c13560571f419e9b6f45edee3a2a37, message={"server":"","servicePrincipal":"","db":"default","table":"table_01c13560571f419e9b6f45edee3a2a37","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_01c13560571f419e9b6f45edee3a2a37\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_01c13560571f419e9b6f45edee3a2a37\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_01c13560571f419e9b6f45edee3a2a37\",\"kudu.table_id\":\"aee290f61c8f4d7b93d4cd7192eb9277\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_a31a093a024346aab31bc8d55a00253d, message={"server":"","servicePrincipal":"","db":"default","table":"table_a31a093a024346aab31bc8d55a00253d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a31a093a024346aab31bc8d55a00253d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_a31a093a024346aab31bc8d55a00253d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a31a093a024346aab31bc8d55a00253d\",\"kudu.table_id\":\"81aee385d1404c82aaeebee220cb9e68\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_60554f72b1aa48aa93aaea511a985fe7, message={"server":"","servicePrincipal":"","db":"default","table":"table_60554f72b1aa48aa93aaea511a985fe7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_60554f72b1aa48aa93aaea511a985fe7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881667},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_60554f72b1aa48aa93aaea511a985fe7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_60554f72b1aa48aa93aaea511a985fe7\",\"kudu.table_id\":\"e46f78618ca648e19c3935d0bf874a35\",\"transient_lastDdlTime\":\"1750881667\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fb46e8f3dcd google::LogMessage::Fail() at ??:0
@ 0x7fb46e8f7b93 google::LogMessage::SendToLog() at ??:0
I20250625 20:01:24.984661 3040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:24.984776 3040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:24.984793 2997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f87f9e18e5b948a4a3fb3666f106abd0. Latest consensus state: current_term: 8 leader_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:24.984913 2997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:24.984846 3040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:24.984985 3040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fb46e8f37cc google::LogMessage::Flush() at ??:0
@ 0x7fb46e8f3a11 google::LogMessage::~LogMessage() at ??:0
I20250625 20:01:24.988837 3032 mvcc.cc:204] Tried to move back new op lower bound from 7171611367142068224 to 7171611358067081216. Current Snapshot: MvccSnapshot[applied={T|T < 7171611367142068224}]
I20250625 20:01:24.989948 3023 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:24.990063 3023 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: LEADER
@ 0x7fb47388cc91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250625 20:01:24.990810 3040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:24.990918 3040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:24.992300 2547 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Sending DeleteTablet for 3 replicas of tablet d4cd08f25a694ab9949c119997cad26d
@ 0x7fb47388bba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250625 20:01:24.993417 2547 catalog_manager.cc:6652] ReplaceTablet: tablet d4cd08f25a694ab9949c119997cad26d deleted and replaced by tablet 912f25c1234a4f3199bd22cb33ea3788
I20250625 20:01:24.993729 2547 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41730:
name: "default.Table_a31a093a024346aab31bc8d55a00253d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fb47388b30b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
@ 0x7fb473890a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56372ba55752 std::function<>::operator()() at ??:0
I20250625 20:01:25.007480 1397 tablet_service.cc:1515] Processing DeleteTablet for tablet d4cd08f25a694ab9949c119997cad26d with delete_type TABLET_DATA_DELETED (replaced by tablet 912f25c1234a4f3199bd22cb33ea3788) from {username='slave'} at 127.0.0.1:55080
I20250625 20:01:25.007870 1533 tablet_service.cc:1515] Processing DeleteTablet for tablet d4cd08f25a694ab9949c119997cad26d with delete_type TABLET_DATA_DELETED (replaced by tablet 912f25c1234a4f3199bd22cb33ea3788) from {username='slave'} at 127.0.0.1:60382
I20250625 20:01:25.008005 1399 tablet_service.cc:1468] Processing CreateTablet for tablet 912f25c1234a4f3199bd22cb33ea3788 (DEFAULT_TABLE table=default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:25.008075 1532 tablet_service.cc:1468] Processing CreateTablet for tablet 912f25c1234a4f3199bd22cb33ea3788 (DEFAULT_TABLE table=default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:25.008302 1399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 912f25c1234a4f3199bd22cb33ea3788. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:25.008358 1532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 912f25c1234a4f3199bd22cb33ea3788. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fb46fd450a1 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb4705346db start_thread at ??:0
I20250625 20:01:25.010735 3050 tablet_replica.cc:331] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:25.010812 1676 tablet_service.cc:1515] Processing DeleteTablet for tablet d4cd08f25a694ab9949c119997cad26d with delete_type TABLET_DATA_DELETED (replaced by tablet 912f25c1234a4f3199bd22cb33ea3788) from {username='slave'} at 127.0.0.1:57992
I20250625 20:01:25.010865 3050 raft_consensus.cc:2241] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:25.011075 3050 raft_consensus.cc:2270] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:25.011267 1675 tablet_service.cc:1468] Processing CreateTablet for tablet 912f25c1234a4f3199bd22cb33ea3788 (DEFAULT_TABLE table=default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:25.011332 3053 tablet_bootstrap.cc:492] T 912f25c1234a4f3199bd22cb33ea3788 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:25.011477 3050 ts_tablet_manager.cc:1905] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:25.011514 1675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 912f25c1234a4f3199bd22cb33ea3788. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fb46daaa71f clone at ??:0
I20250625 20:01:25.012687 3053 tablet_bootstrap.cc:654] T 912f25c1234a4f3199bd22cb33ea3788 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:25.013808 3051 tablet_replica.cc:331] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:25.013813 3050 ts_tablet_manager.cc:1918] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:25.013870 3050 log.cc:1199] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/d4cd08f25a694ab9949c119997cad26d
I20250625 20:01:25.013934 3051 raft_consensus.cc:2241] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:25.014057 3051 raft_consensus.cc:2270] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:25.014086 3050 ts_tablet_manager.cc:1939] T d4cd08f25a694ab9949c119997cad26d P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:25.014298 3055 tablet_replica.cc:331] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:25.014407 3051 ts_tablet_manager.cc:1905] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:25.014426 3055 raft_consensus.cc:2241] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:25.014528 3055 raft_consensus.cc:2270] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:25.015003 3055 ts_tablet_manager.cc:1905] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:25.015923 3054 tablet_bootstrap.cc:492] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:25.016431 3051 ts_tablet_manager.cc:1918] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:25.016495 3051 log.cc:1199] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/d4cd08f25a694ab9949c119997cad26d
I20250625 20:01:25.017010 3051 ts_tablet_manager.cc:1939] T d4cd08f25a694ab9949c119997cad26d P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:25.017105 3054 tablet_bootstrap.cc:654] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:25.017156 3055 ts_tablet_manager.cc:1918] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:25.017216 3055 log.cc:1199] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/d4cd08f25a694ab9949c119997cad26d
I20250625 20:01:25.017433 3055 ts_tablet_manager.cc:1939] T d4cd08f25a694ab9949c119997cad26d P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:25.018143 3054 tablet_bootstrap.cc:492] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:25.018217 3054 ts_tablet_manager.cc:1397] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250625 20:01:25.018395 3057 tablet_bootstrap.cc:492] T 912f25c1234a4f3199bd22cb33ea3788 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:25.018563 3053 tablet_bootstrap.cc:492] T 912f25c1234a4f3199bd22cb33ea3788 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:25.018635 3053 ts_tablet_manager.cc:1397] T 912f25c1234a4f3199bd22cb33ea3788 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250625 20:01:25.018683 3054 raft_consensus.cc:357] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:25.018806 3054 raft_consensus.cc:383] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:25.018838 3054 raft_consensus.cc:738] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:25.018975 3054 consensus_queue.cc:260] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:25.019083 3053 raft_consensus.cc:357] T 912f25c1234a4f3199bd22cb33ea3788 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:25.019181 3053 raft_consensus.cc:383] T 912f25c1234a4f3199bd22cb33ea3788 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:25.019212 3053 raft_consensus.cc:738] T 912f25c1234a4f3199bd22cb33ea3788 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:25.019254 3054 ts_tablet_manager.cc:1428] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:25.019332 3053 consensus_queue.cc:260] T 912f25c1234a4f3199bd22cb33ea3788 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:25.019392 3057 tablet_bootstrap.cc:654] T 912f25c1234a4f3199bd22cb33ea3788 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:25.019591 3053 ts_tablet_manager.cc:1428] T 912f25c1234a4f3199bd22cb33ea3788 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 20:01:25.021988 3057 tablet_bootstrap.cc:492] T 912f25c1234a4f3199bd22cb33ea3788 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:25.022075 3057 ts_tablet_manager.cc:1397] T 912f25c1234a4f3199bd22cb33ea3788 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250625 20:01:25.022575 3057 raft_consensus.cc:357] T 912f25c1234a4f3199bd22cb33ea3788 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:25.022675 3057 raft_consensus.cc:383] T 912f25c1234a4f3199bd22cb33ea3788 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:25.022704 3057 raft_consensus.cc:738] T 912f25c1234a4f3199bd22cb33ea3788 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:25.022835 3057 consensus_queue.cc:260] T 912f25c1234a4f3199bd22cb33ea3788 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:25.023296 3057 ts_tablet_manager.cc:1428] T 912f25c1234a4f3199bd22cb33ea3788 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:25.094236 2720 raft_consensus.cc:491] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:25.094373 2720 raft_consensus.cc:513] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:25.094933 2720 leader_election.cc:290] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:25.095242 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "912f25c1234a4f3199bd22cb33ea3788" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:25.095269 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "912f25c1234a4f3199bd22cb33ea3788" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:25.095371 1700 raft_consensus.cc:2466] T 912f25c1234a4f3199bd22cb33ea3788 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:25.095407 1560 raft_consensus.cc:2466] T 912f25c1234a4f3199bd22cb33ea3788 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:25.095638 1358 leader_election.cc:304] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:25.095829 2720 raft_consensus.cc:2802] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:25.095898 2720 raft_consensus.cc:491] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:25.095950 2720 raft_consensus.cc:3058] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:25.097100 2720 raft_consensus.cc:513] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:25.097479 2720 leader_election.cc:290] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:25.097790 1700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "912f25c1234a4f3199bd22cb33ea3788" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:25.097805 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "912f25c1234a4f3199bd22cb33ea3788" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:25.097898 1700 raft_consensus.cc:3058] T 912f25c1234a4f3199bd22cb33ea3788 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:25.097918 1560 raft_consensus.cc:3058] T 912f25c1234a4f3199bd22cb33ea3788 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:25.098636 1700 raft_consensus.cc:2466] T 912f25c1234a4f3199bd22cb33ea3788 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:25.098867 1560 raft_consensus.cc:2466] T 912f25c1234a4f3199bd22cb33ea3788 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:25.098898 1358 leader_election.cc:304] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:25.099066 2720 raft_consensus.cc:2802] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:25.099244 2720 raft_consensus.cc:695] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:25.099376 2720 consensus_queue.cc:237] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
W20250625 20:01:25.166422 1494 connection.cc:537] server connection from 127.0.0.1:60382 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:25.166422 1635 connection.cc:537] server connection from 127.0.0.1:57992 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:25.166721 1357 connection.cc:537] client connection to 127.0.204.61:38005 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:25.166908 1357 connection.cc:537] server connection from 127.0.0.1:55080 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:25.167162 1634 connection.cc:537] client connection to 127.0.204.61:38005 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:25.167186 1494 connection.cc:537] client connection to 127.0.204.61:38005 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:25.168279 3017 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 20:01:25.167796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.204.61:38005 (local address 127.0.204.2:47581)
0625 20:01:25.167993 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0625 20:01:25.168007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0625 20:01:25.168090 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 20:01:25.168163 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":107}
W20250625 20:01:25.168483 1754 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.204.61:38005)
W20250625 20:01:25.168797 1754 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.204.61:38005)
W20250625 20:01:25.168871 3018 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 20:01:25.167599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.204.61:38005 (local address 127.0.204.1:57759)
0625 20:01:25.167854 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0625 20:01:25.167871 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0625 20:01:25.167971 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 20:01:25.168770 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":155}
W20250625 20:01:25.168892 1754 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.204.61:38005)
W20250625 20:01:25.168972 1754 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.204.61:38005)
I20250625 20:01:25.500031 3070 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f87f9e18e5b948a4a3fb3666f106abd0)
I20250625 20:01:25.500159 3070 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:25.500579 3070 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:25.501648 2941 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
W20250625 20:01:25.501735 2944 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:25.501827 2944 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:25.501998 3070 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250625 20:01:25.641283 2720 consensus_queue.cc:1035] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:25.644707 2720 consensus_queue.cc:1035] T 912f25c1234a4f3199bd22cb33ea3788 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:26.118068 3084 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f87f9e18e5b948a4a3fb3666f106abd0)
I20250625 20:01:26.118223 3084 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:26.118762 3084 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:26.120128 2941 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
W20250625 20:01:26.120235 2944 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:26.120296 2944 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:26.120414 3084 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250625 20:01:26.911959 816 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu with pid 2924
I20250625 20:01:27.457724 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.62:36889
--webserver_interface=127.0.204.62
--webserver_port=37479
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 20:01:27.563057 3104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:01:27.563294 3104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:27.563335 3104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:01:27.566537 3104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:01:27.566605 3104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:27.566625 3104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:01:27.566642 3104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:01:27.570406 3104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.62:36889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.204.62
--webserver_port=37479
--never_fsync=true
--heap_profile_path=/tmp/kudu.3104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:27.571491 3104 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:27.572679 3104 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:01:27.579612 3110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:27.579670 3109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:27.579588 3112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:27.580107 3104 server_base.cc:1048] running on GCE node
I20250625 20:01:27.580579 3104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:27.581461 3104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:27.582623 3104 hybrid_clock.cc:648] HybridClock initialized: now 1750881687582598 us; error 40 us; skew 500 ppm
I20250625 20:01:27.584728 3104 webserver.cc:469] Webserver started at http://127.0.204.62:37479/ using document root <none> and password file <none>
I20250625 20:01:27.585343 3104 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:27.585400 3104 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:27.589466 3104 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20250625 20:01:27.591528 3118 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:27.592626 3104 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250625 20:01:27.592738 3104 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
uuid: "5852fb53503b438c92f294e0c44c916d"
format_stamp: "Formatted at 2025-06-25 20:01:04 on dist-test-slave-lc57"
I20250625 20:01:27.593161 3104 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:27.605342 3104 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:27.605984 3104 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:27.606153 3104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:27.613016 3170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.62:36889 every 8 connection(s)
I20250625 20:01:27.613013 3104 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.62:36889
I20250625 20:01:27.614223 3104 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/info.pb
I20250625 20:01:27.617908 3171 sys_catalog.cc:263] Verifying existing consensus state
I20250625 20:01:27.619488 3171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Bootstrap starting.
I20250625 20:01:27.623435 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 3104
I20250625 20:01:27.631461 3171 log.cc:826] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:27.654289 1748 heartbeater.cc:344] Connected to a master server at 127.0.204.62:36889
I20250625 20:01:27.657672 1608 heartbeater.cc:344] Connected to a master server at 127.0.204.62:36889
I20250625 20:01:27.684620 3171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Bootstrap replayed 1/1 log segments. Stats: ops{read=110 overwritten=1 applied=109 ignored=0} inserts{seen=72 ignored=0} mutations{seen=98 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:01:27.685002 3171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Bootstrap complete.
I20250625 20:01:27.687562 3171 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:27.687908 3171 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5852fb53503b438c92f294e0c44c916d, State: Initialized, Role: FOLLOWER
I20250625 20:01:27.688351 3171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 109, Last appended: 8.109, Last appended by leader: 109, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:27.688959 3178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:27.689065 3178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:27.689373 3171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:01:27.696065 3192 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:27.696162 3192 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: loading cluster ID for follower catalog manager: success
I20250625 20:01:27.699285 3192 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: acquiring CA information for follower catalog manager: success
I20250625 20:01:27.701601 3192 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 20:01:27.708072 3104 master_runner.cc:186] Error getting master registration for 127.0.204.61:38005: Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111),
I20250625 20:01:27.710083 3104 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:27.762907 2897 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:27.763056 2897 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:27.763619 2897 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:01:27.764366 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:01:27.764492 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:01:27.764583 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:01:27.764725 2897 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:01:27.773296 1471 heartbeater.cc:344] Connected to a master server at 127.0.204.62:36889
I20250625 20:01:27.774477 3135 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" instance_seqno: 1750881665574847) as {username='slave'} at 127.0.204.1:41567; Asking this server to re-register.
I20250625 20:01:27.774888 1471 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:27.776075 3135 ts_manager.cc:194] Registered new tserver with Master: 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:28.157395 3135 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" instance_seqno: 1750881665979916) as {username='slave'} at 127.0.204.3:60651; Asking this server to re-register.
I20250625 20:01:28.157852 1748 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:28.158807 3135 ts_manager.cc:194] Registered new tserver with Master: 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:28.160588 3135 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c73846c0665d4d53a274d053946107df" instance_seqno: 1750881665791183) as {username='slave'} at 127.0.204.2:40427; Asking this server to re-register.
I20250625 20:01:28.160951 1608 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:28.161806 3135 ts_manager.cc:194] Registered new tserver with Master: c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:28.284533 3201 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:28.284747 3201 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:28.285959 3201 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:28.286531 3122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:01:28.287252 3119 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
W20250625 20:01:28.287372 3122 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:28.287441 3122 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:28.287637 3201 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250625 20:01:28.972777 3211 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:28.972906 3211 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:28.973366 3211 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:28.974413 3119 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
W20250625 20:01:28.974525 3122 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:28.974601 3122 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:28.974723 3211 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250625 20:01:29.813889 3224 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:29.814026 3224 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:29.814435 3224 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:29.815506 3119 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
W20250625 20:01:29.815629 3122 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:29.815699 3122 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:29.815850 3224 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250625 20:01:30.464769 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.60:46191
--webserver_interface=127.0.204.60
--webserver_port=43997
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 20:01:30.508100 3236 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:30.508239 3236 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:30.508684 3236 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:30.509936 3119 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
W20250625 20:01:30.510037 3122 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:30.510095 3122 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:30.510231 3236 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250625 20:01:30.569622 3235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:01:30.569834 3235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:30.569890 3235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:01:30.573043 3235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:01:30.573101 3235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:30.573128 3235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:01:30.573151 3235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:01:30.576882 3235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.60:46191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.204.60
--webserver_port=43997
--never_fsync=true
--heap_profile_path=/tmp/kudu.3235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:30.577956 3235 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:30.579061 3235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:01:30.585850 3245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:30.585860 3247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:30.585845 3244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:30.586089 3235 server_base.cc:1048] running on GCE node
I20250625 20:01:30.586701 3235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:30.587807 3235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:30.588966 3235 hybrid_clock.cc:648] HybridClock initialized: now 1750881690588944 us; error 37 us; skew 500 ppm
I20250625 20:01:30.596863 3235 webserver.cc:469] Webserver started at http://127.0.204.60:43997/ using document root <none> and password file <none>
I20250625 20:01:30.597498 3235 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:30.597576 3235 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:30.601287 3235 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20250625 20:01:30.603319 3253 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:30.604472 3235 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250625 20:01:30.604573 3235 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
uuid: "1fcdee0c0aac4013a5e3c77304b474d9"
format_stamp: "Formatted at 2025-06-25 20:01:05 on dist-test-slave-lc57"
I20250625 20:01:30.604962 3235 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:30.618790 3235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:30.619437 3235 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:30.619586 3235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:30.626516 3235 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.60:46191
I20250625 20:01:30.626536 3305 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.60:46191 every 8 connection(s)
I20250625 20:01:30.627696 3235 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-2/data/info.pb
I20250625 20:01:30.630613 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 3235
I20250625 20:01:30.631479 3306 sys_catalog.cc:263] Verifying existing consensus state
I20250625 20:01:30.633495 3306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Bootstrap starting.
I20250625 20:01:30.644945 3306 log.cc:826] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:30.660777 1607 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:30.661520 1747 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:30.697213 3306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=106 overwritten=0 applied=106 ignored=0} inserts{seen=71 ignored=0} mutations{seen=96 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:01:30.697603 3306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Bootstrap complete.
I20250625 20:01:30.701236 3306 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:30.701663 3306 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcdee0c0aac4013a5e3c77304b474d9, State: Initialized, Role: FOLLOWER
I20250625 20:01:30.702193 3306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 106, Last appended: 7.106, Last appended by leader: 106, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:30.702687 3316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:30.702819 3316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:30.703476 3306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:01:30.711349 3330 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:30.711436 3330 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: loading cluster ID for follower catalog manager: success
I20250625 20:01:30.714814 3330 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: acquiring CA information for follower catalog manager: success
I20250625 20:01:30.716988 3330 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 20:01:30.720616 3235 master_runner.cc:186] Error getting master registration for 127.0.204.61:38005: Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111),
I20250625 20:01:30.722946 3235 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:30.778822 1470 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:30.780485 3269 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" instance_seqno: 1750881665574847) as {username='slave'} at 127.0.204.1:39849; Asking this server to re-register.
I20250625 20:01:30.781014 1470 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:30.782719 3269 ts_manager.cc:194] Registered new tserver with Master: 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:31.164805 1747 heartbeater.cc:344] Connected to a master server at 127.0.204.60:46191
I20250625 20:01:31.165143 3269 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c73846c0665d4d53a274d053946107df" instance_seqno: 1750881665791183) as {username='slave'} at 127.0.204.2:51545; Asking this server to re-register.
I20250625 20:01:31.165596 1607 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:31.166219 3269 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" instance_seqno: 1750881665979916) as {username='slave'} at 127.0.204.3:58663; Asking this server to re-register.
I20250625 20:01:31.166682 1747 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:31.166682 3270 ts_manager.cc:194] Registered new tserver with Master: c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:31.167656 3270 ts_manager.cc:194] Registered new tserver with Master: 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:31.185351 3316 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:31.185567 3316 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:31.187137 3316 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
W20250625 20:01:31.187623 3254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:31.187614 3145 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" candidate_term: 8 candidate_status { last_received { term: 7 index: 106 } } ignore_live_leader: false dest_uuid: "5852fb53503b438c92f294e0c44c916d" is_pre_election: true
I20250625 20:01:31.187911 3145 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fcdee0c0aac4013a5e3c77304b474d9 in current term 8: Already voted for candidate f87f9e18e5b948a4a3fb3666f106abd0 in this term.
W20250625 20:01:31.188501 3254 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:31.188588 3254 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9; no voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:31.188830 3316 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 7 FOLLOWER]: Advancing to term 8
I20250625 20:01:31.190824 3316 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250625 20:01:31.865005 3342 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:31.865131 3342 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:31.865597 3342 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
I20250625 20:01:31.866024 3145 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" candidate_term: 9 candidate_status { last_received { term: 7 index: 106 } } ignore_live_leader: false dest_uuid: "5852fb53503b438c92f294e0c44c916d" is_pre_election: true
I20250625 20:01:31.866220 3145 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fcdee0c0aac4013a5e3c77304b474d9 for term 9 because replica has last-logged OpId of term: 8 index: 109, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 106.
W20250625 20:01:31.866858 3254 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:31.866930 3254 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9; no voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:31.867080 3342 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250625 20:01:32.405629 3349 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:32.405767 3349 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:32.406168 3349 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:32.407380 3119 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:32.409984 3280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 9 candidate_status { last_received { term: 8 index: 109 } } ignore_live_leader: false dest_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" is_pre_election: true
I20250625 20:01:32.410246 3280 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5852fb53503b438c92f294e0c44c916d in term 8.
I20250625 20:01:32.410566 3122 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d; no voters: f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:32.410749 3349 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250625 20:01:32.410822 3349 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:32.410861 3349 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 8 FOLLOWER]: Advancing to term 9
I20250625 20:01:32.412845 3349 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:32.413246 3349 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 election: Requested vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:32.413664 3280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 9 candidate_status { last_received { term: 8 index: 109 } } ignore_live_leader: false dest_uuid: "1fcdee0c0aac4013a5e3c77304b474d9"
I20250625 20:01:32.413796 3280 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 8 FOLLOWER]: Advancing to term 9
W20250625 20:01:32.414222 3119 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:32.414785 3280 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5852fb53503b438c92f294e0c44c916d in term 9.
I20250625 20:01:32.415045 3122 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d; no voters: f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:32.415198 3349 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 9 FOLLOWER]: Leader election won for term 9
I20250625 20:01:32.415378 3349 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 9 LEADER]: Becoming Leader. State: Replica: 5852fb53503b438c92f294e0c44c916d, State: Running, Role: LEADER
I20250625 20:01:32.415693 3349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 8.109, Last appended by leader: 109, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:32.417357 3354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5852fb53503b438c92f294e0c44c916d. Latest consensus state: current_term: 9 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:32.417465 3354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:32.417745 3356 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:01:32.419492 3356 catalog_manager.cc:671] Loaded metadata for table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]
I20250625 20:01:32.419977 3356 catalog_manager.cc:671] Loaded metadata for table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]
I20250625 20:01:32.420176 3356 catalog_manager.cc:671] Loaded metadata for table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]
I20250625 20:01:32.420363 3356 catalog_manager.cc:671] Loaded metadata for table default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4]
I20250625 20:01:32.420547 3356 catalog_manager.cc:671] Loaded metadata for table default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]
I20250625 20:01:32.420751 3356 catalog_manager.cc:671] Loaded metadata for table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]
I20250625 20:01:32.420949 3356 catalog_manager.cc:671] Loaded metadata for table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996]
I20250625 20:01:32.421132 3356 catalog_manager.cc:671] Loaded metadata for table default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5]
I20250625 20:01:32.421459 3356 catalog_manager.cc:671] Loaded metadata for table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583]
I20250625 20:01:32.421739 3356 catalog_manager.cc:671] Loaded metadata for table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277]
I20250625 20:01:32.422026 3356 catalog_manager.cc:671] Loaded metadata for table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]
I20250625 20:01:32.422209 3356 catalog_manager.cc:671] Loaded metadata for table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f]
I20250625 20:01:32.422389 3356 catalog_manager.cc:671] Loaded metadata for table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]
I20250625 20:01:32.422574 3356 catalog_manager.cc:671] Loaded metadata for table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]
I20250625 20:01:32.422771 3356 catalog_manager.cc:671] Loaded metadata for table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]
I20250625 20:01:32.424343 3356 tablet_loader.cc:96] loaded metadata for tablet 01f62ece8947463daef2b44571d4dec0 (table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e])
I20250625 20:01:32.424552 3356 tablet_loader.cc:96] loaded metadata for tablet 06ef5241d21449169f228cfa37ff91b4 (table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996])
I20250625 20:01:32.425204 3356 tablet_loader.cc:96] loaded metadata for tablet 3dd7cedb1c3542c086ebae940840147b (table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8])
I20250625 20:01:32.425582 3356 tablet_loader.cc:96] loaded metadata for tablet 6b660df5bca642af848d7464859ac7fc (table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa])
I20250625 20:01:32.425736 3356 tablet_loader.cc:96] loaded metadata for tablet 6c8e829c96474e94a8de9e7e32b48350 (table default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4])
I20250625 20:01:32.426049 3356 tablet_loader.cc:96] loaded metadata for tablet 78631bcca66d4b16b344e57f8a1cc291 (table default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5])
I20250625 20:01:32.426193 3356 tablet_loader.cc:96] loaded metadata for tablet 7b1d8f876f744c868857b56e49dda3e7 (table default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7])
I20250625 20:01:32.426416 3356 tablet_loader.cc:96] loaded metadata for tablet 912f25c1234a4f3199bd22cb33ea3788 (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809])
I20250625 20:01:32.427091 3356 tablet_loader.cc:96] loaded metadata for tablet bb6c755fce6640d2a0b13a449d3adfb9 (table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583])
I20250625 20:01:32.427234 3356 tablet_loader.cc:96] loaded metadata for tablet c00e8f8653bd42fda7bef627e99930cc (table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f])
I20250625 20:01:32.427531 3356 tablet_loader.cc:96] loaded metadata for tablet cdb2a543898642d3abc3afa850178120 (table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277])
I20250625 20:01:32.427667 3356 tablet_loader.cc:96] loaded metadata for tablet d1b4020c408349719f7533e4b6c0d1cd (table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35])
I20250625 20:01:32.428087 3356 tablet_loader.cc:96] loaded metadata for tablet d8dd4c0e80974a67b63293d83760fd8b (table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36])
I20250625 20:01:32.428390 3356 tablet_loader.cc:96] loaded metadata for tablet e44908760f6245a282f72cec3b5773fe (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0])
I20250625 20:01:32.428531 3356 tablet_loader.cc:96] loaded metadata for tablet eb717c706f3d428bbd5bf6c8e4f839aa (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1])
I20250625 20:01:32.428660 3356 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:01:32.429400 3356 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:32.429446 3356 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:01:32.430287 3356 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:01:32.430948 3356 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Loaded TSK: 0
I20250625 20:01:32.431663 3356 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:01:32.432345 3356 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 20:01:32.433003 3356 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 21
I20250625 20:01:32.474699 3135 master_service.cc:948] ReplaceTablet: received request to replace tablet d1b4020c408349719f7533e4b6c0d1cd from {username='slave'} at 127.0.0.1:52972
I20250625 20:01:32.474804 3129 master_service.cc:948] ReplaceTablet: received request to replace tablet 7b1d8f876f744c868857b56e49dda3e7 from {username='slave'} at 127.0.0.1:52972
I20250625 20:01:32.474738 3134 master_service.cc:948] ReplaceTablet: received request to replace tablet 6c8e829c96474e94a8de9e7e32b48350 from {username='slave'} at 127.0.0.1:52972
I20250625 20:01:32.475426 3133 master_service.cc:948] ReplaceTablet: received request to replace tablet 7b1d8f876f744c868857b56e49dda3e7 from {username='slave'} at 127.0.0.1:52972
I20250625 20:01:32.475654 3132 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52972:
table { table_name: "default.Table_fd28d4c5b48245c382a334b98a3066df" } modify_external_catalogs: true
I20250625 20:01:32.475813 3132 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52972:
table { table_name: "default.Table_fd28d4c5b48245c382a334b98a3066df" } modify_external_catalogs: true
I20250625 20:01:32.476156 3128 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52972:
name: "default.Table_117db1e3446d41099601e6b995a36d74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:32.475373 3126 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52972:
table { table_name: "default.Table_976af21482a74bc79eb800684cce978e" } new_table_name: "default.Table_ad0e628a3e004a9da4a42cb7d6051e3e" modify_external_catalogs: true
I20250625 20:01:32.476555 3127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52972:
name: "default.Table_e91ea2924c1b40fc9d7610784c293467"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:32.476778 3130 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52972:
name: "default.Table_ea337ae9a0e74819be0ba8940e1828e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:32.476949 3131 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52972:
table { table_name: "default.Table_1d0eb88cbdfd4b16bb66bfb3d502672b" } new_table_name: "default.Table_dd960e4b8a9c42309150252433773c76" modify_external_catalogs: true
I20250625 20:01:32.478663 3280 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 9 FOLLOWER]: Refusing update from remote peer 5852fb53503b438c92f294e0c44c916d: Log matching property violated. Preceding OpId in replica: term: 7 index: 106. Preceding OpId from leader: term: 9 index: 111. (index mismatch)
I20250625 20:01:32.479218 3354 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 106, Time since last communication: 0.000s
W20250625 20:01:32.482143 3119 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d -> Peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Couldn't send request to peer f87f9e18e5b948a4a3fb3666f106abd0. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20250625 20:01:32.482746 3191 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 6 out of 13 received events: NotificationEvent(eventId=22, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_063972c2ee4b4e468cf982177f09f91a, message={"server":"","servicePrincipal":"","db":"default","table":"table_063972c2ee4b4e468cf982177f09f91a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_063972c2ee4b4e468cf982177f09f91a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_063972c2ee4b4e468cf982177f09f91a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_063972c2ee4b4e468cf982177f09f91a\",\"kudu.table_id\":\"a054c42074394f1db21dfcf2c55d99b5\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_01c13560571f419e9b6f45edee3a2a37, message={"server":"","servicePrincipal":"","db":"default","table":"table_01c13560571f419e9b6f45edee3a2a37","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_01c13560571f419e9b6f45edee3a2a37\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_01c13560571f419e9b6f45edee3a2a37\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_01c13560571f419e9b6f45edee3a2a37\",\"kudu.table_id\":\"aee290f61c8f4d7b93d4cd7192eb9277\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_a31a093a024346aab31bc8d55a00253d, message={"server":"","servicePrincipal":"","db":"default","table":"table_a31a093a024346aab31bc8d55a00253d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a31a093a024346aab31bc8d55a00253d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_a31a093a024346aab31bc8d55a00253d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a31a093a024346aab31bc8d55a00253d\",\"kudu.table_id\":\"81aee385d1404c82aaeebee220cb9e68\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_60554f72b1aa48aa93aaea511a985fe7, message={"server":"","servicePrincipal":"","db":"default","table":"table_60554f72b1aa48aa93aaea511a985fe7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_60554f72b1aa48aa93aaea511a985fe7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881667},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_60554f72b1aa48aa93aaea511a985fe7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_60554f72b1aa48aa93aaea511a985fe7\",\"kudu.table_id\":\"e46f78618ca648e19c3935d0bf874a35\",\"transient_lastDdlTime\":\"1750881667\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250625 20:01:32.484581 3361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5852fb53503b438c92f294e0c44c916d. Latest consensus state: current_term: 9 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:32.485353 3361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:32.485491 3361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:32.485621 3361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fa97bf86dcd google::LogMessage::Fail() at ??:0
I20250625 20:01:32.487231 3349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:32.487327 3349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:32.487660 3358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:32.487772 3358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:32.488745 3135 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Sending DeleteTablet for 3 replicas of tablet d1b4020c408349719f7533e4b6c0d1cd
@ 0x7fa97bf8ab93 google::LogMessage::SendToLog() at ??:0
I20250625 20:01:32.489935 3135 catalog_manager.cc:6652] ReplaceTablet: tablet d1b4020c408349719f7533e4b6c0d1cd deleted and replaced by tablet 68a1a119b2b448afabe18cc28a474f4c
I20250625 20:01:32.489967 3359 mvcc.cc:204] Tried to move back new op lower bound from 7171611412393238528 to 7171611381411131392. Current Snapshot: MvccSnapshot[applied={T|T < 7171611381743222784}]
I20250625 20:01:32.490196 3135 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52972:
table { table_name: "default.Table_60554f72b1aa48aa93aaea511a985fe7" } modify_external_catalogs: true
I20250625 20:01:32.490301 3135 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52972:
table { table_name: "default.Table_60554f72b1aa48aa93aaea511a985fe7" } modify_external_catalogs: true
I20250625 20:01:32.490530 3357 mvcc.cc:204] Tried to move back new op lower bound from 7171611412393238528 to 7171611412137127936. Current Snapshot: MvccSnapshot[applied={T|T < 7171611412391575552}]
@ 0x7fa97bf867cc google::LogMessage::Flush() at ??:0
I20250625 20:01:32.492506 3134 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Sending DeleteTablet for 3 replicas of tablet 6c8e829c96474e94a8de9e7e32b48350
I20250625 20:01:32.492786 3134 catalog_manager.cc:6652] ReplaceTablet: tablet 6c8e829c96474e94a8de9e7e32b48350 deleted and replaced by tablet a90d15fb49c94d3e85a5ba26aab2878a
I20250625 20:01:32.492918 3129 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Sending DeleteTablet for 3 replicas of tablet 7b1d8f876f744c868857b56e49dda3e7
I20250625 20:01:32.492964 3134 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52972:
table { table_name: "default.Table_063972c2ee4b4e468cf982177f09f91a" } new_table_name: "default.Table_980a5f7891e24ccc87c72a858cc5076f" modify_external_catalogs: true
@ 0x7fa97bf86a11 google::LogMessage::~LogMessage() at ??:0
I20250625 20:01:32.493170 3129 catalog_manager.cc:6652] ReplaceTablet: tablet 7b1d8f876f744c868857b56e49dda3e7 deleted and replaced by tablet b927d1839a3b4b2690cabac2e19f74d5
I20250625 20:01:32.493311 3129 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52972:
table { table_name: "default.Table_01c13560571f419e9b6f45edee3a2a37" } modify_external_catalogs: true
I20250625 20:01:32.493295 3133 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52972:
name: "default.Table_3629c083428e4ae0a977ffb16af83bc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:32.493394 3129 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52972:
table { table_name: "default.Table_01c13560571f419e9b6f45edee3a2a37" } modify_external_catalogs: true
@ 0x7fa980f1fc91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250625 20:01:32.500605 1675 tablet_service.cc:1515] Processing DeleteTablet for tablet d1b4020c408349719f7533e4b6c0d1cd with delete_type TABLET_DATA_DELETED (replaced by tablet 68a1a119b2b448afabe18cc28a474f4c) from {username='slave'} at 127.0.0.1:58004
I20250625 20:01:32.500857 1676 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c8e829c96474e94a8de9e7e32b48350 with delete_type TABLET_DATA_DELETED (replaced by tablet a90d15fb49c94d3e85a5ba26aab2878a) from {username='slave'} at 127.0.0.1:58004
I20250625 20:01:32.500988 1673 tablet_service.cc:1515] Processing DeleteTablet for tablet 7b1d8f876f744c868857b56e49dda3e7 with delete_type TABLET_DATA_DELETED (replaced by tablet b927d1839a3b4b2690cabac2e19f74d5) from {username='slave'} at 127.0.0.1:58004
@ 0x7fa980f1eba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250625 20:01:32.504218 1673 tablet_service.cc:1468] Processing CreateTablet for tablet 68a1a119b2b448afabe18cc28a474f4c (DEFAULT_TABLE table=default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:32.504505 1673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68a1a119b2b448afabe18cc28a474f4c. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fa980f1e30b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250625 20:01:32.507274 3374 tablet_replica.cc:331] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:32.507400 3374 raft_consensus.cc:2241] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:32.507580 3374 raft_consensus.cc:2270] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:32.507993 3374 ts_tablet_manager.cc:1905] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:32.508708 1673 tablet_service.cc:1468] Processing CreateTablet for tablet a90d15fb49c94d3e85a5ba26aab2878a (DEFAULT_TABLE table=default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:32.508946 1673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a90d15fb49c94d3e85a5ba26aab2878a. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fa980f23a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250625 20:01:32.510710 1399 tablet_service.cc:1515] Processing DeleteTablet for tablet d1b4020c408349719f7533e4b6c0d1cd with delete_type TABLET_DATA_DELETED (replaced by tablet 68a1a119b2b448afabe18cc28a474f4c) from {username='slave'} at 127.0.0.1:55088
I20250625 20:01:32.510762 1401 tablet_service.cc:1468] Processing CreateTablet for tablet 68a1a119b2b448afabe18cc28a474f4c (DEFAULT_TABLE table=default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:32.510804 1400 tablet_service.cc:1468] Processing CreateTablet for tablet a90d15fb49c94d3e85a5ba26aab2878a (DEFAULT_TABLE table=default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:32.510950 1397 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c8e829c96474e94a8de9e7e32b48350 with delete_type TABLET_DATA_DELETED (replaced by tablet a90d15fb49c94d3e85a5ba26aab2878a) from {username='slave'} at 127.0.0.1:55088
I20250625 20:01:32.511004 1401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68a1a119b2b448afabe18cc28a474f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:32.511057 1398 tablet_service.cc:1515] Processing DeleteTablet for tablet 7b1d8f876f744c868857b56e49dda3e7 with delete_type TABLET_DATA_DELETED (replaced by tablet b927d1839a3b4b2690cabac2e19f74d5) from {username='slave'} at 127.0.0.1:55088
I20250625 20:01:32.511142 1400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a90d15fb49c94d3e85a5ba26aab2878a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:32.511358 1402 tablet_service.cc:1468] Processing CreateTablet for tablet b927d1839a3b4b2690cabac2e19f74d5 (DEFAULT_TABLE table=default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:32.511487 1676 tablet_service.cc:1468] Processing CreateTablet for tablet b927d1839a3b4b2690cabac2e19f74d5 (DEFAULT_TABLE table=default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:32.511562 1402 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b927d1839a3b4b2690cabac2e19f74d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:32.511673 1676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b927d1839a3b4b2690cabac2e19f74d5. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x5572d3d8a752 std::function<>::operator()() at ??:0
I20250625 20:01:32.516988 3374 ts_tablet_manager.cc:1918] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:32.517063 3374 log.cc:1199] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/d1b4020c408349719f7533e4b6c0d1cd
I20250625 20:01:32.517290 3374 ts_tablet_manager.cc:1939] T d1b4020c408349719f7533e4b6c0d1cd P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:32.517345 1532 tablet_service.cc:1515] Processing DeleteTablet for tablet d1b4020c408349719f7533e4b6c0d1cd with delete_type TABLET_DATA_DELETED (replaced by tablet 68a1a119b2b448afabe18cc28a474f4c) from {username='slave'} at 127.0.0.1:60394
I20250625 20:01:32.517505 1536 tablet_service.cc:1468] Processing CreateTablet for tablet a90d15fb49c94d3e85a5ba26aab2878a (DEFAULT_TABLE table=default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:32.517583 1533 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c8e829c96474e94a8de9e7e32b48350 with delete_type TABLET_DATA_DELETED (replaced by tablet a90d15fb49c94d3e85a5ba26aab2878a) from {username='slave'} at 127.0.0.1:60394
I20250625 20:01:32.517696 1537 tablet_service.cc:1515] Processing DeleteTablet for tablet 7b1d8f876f744c868857b56e49dda3e7 with delete_type TABLET_DATA_DELETED (replaced by tablet b927d1839a3b4b2690cabac2e19f74d5) from {username='slave'} at 127.0.0.1:60394
I20250625 20:01:32.517760 1536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a90d15fb49c94d3e85a5ba26aab2878a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:32.517910 1538 tablet_service.cc:1468] Processing CreateTablet for tablet 68a1a119b2b448afabe18cc28a474f4c (DEFAULT_TABLE table=default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:32.517954 3120 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet d1b4020c408349719f7533e4b6c0d1cd (table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]) successfully deleted
I20250625 20:01:32.518128 1538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68a1a119b2b448afabe18cc28a474f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:32.518203 3378 tablet_replica.cc:331] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:32.518323 3378 raft_consensus.cc:2241] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:32.518427 3378 raft_consensus.cc:2270] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:32.518798 3378 ts_tablet_manager.cc:1905] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:32.520116 1534 tablet_service.cc:1468] Processing CreateTablet for tablet b927d1839a3b4b2690cabac2e19f74d5 (DEFAULT_TABLE table=default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:32.520340 1534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b927d1839a3b4b2690cabac2e19f74d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:32.520728 3378 ts_tablet_manager.cc:1918] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:32.520788 3378 log.cc:1199] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/d1b4020c408349719f7533e4b6c0d1cd
I20250625 20:01:32.521040 3378 ts_tablet_manager.cc:1939] T d1b4020c408349719f7533e4b6c0d1cd P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
@ 0x7fa97d3d80a1 kudu::Thread::SuperviseThread() at ??:0
I20250625 20:01:32.521672 3122 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet d1b4020c408349719f7533e4b6c0d1cd (table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]) successfully deleted
I20250625 20:01:32.521981 3377 tablet_bootstrap.cc:492] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:32.522929 3374 tablet_replica.cc:331] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:32.522994 3377 tablet_bootstrap.cc:654] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:32.523051 3374 raft_consensus.cc:2241] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:32.523319 3374 raft_consensus.cc:2270] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:32.523495 3380 tablet_replica.cc:331] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:32.523618 3380 raft_consensus.cc:2241] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:32.523703 3374 ts_tablet_manager.cc:1905] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:32.523741 3380 raft_consensus.cc:2270] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7fa97dbc76db start_thread at ??:0
I20250625 20:01:32.524093 3380 ts_tablet_manager.cc:1905] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fa97b13d71f clone at ??:0
I20250625 20:01:32.525782 3374 ts_tablet_manager.cc:1918] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:32.525847 3374 log.cc:1199] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/6c8e829c96474e94a8de9e7e32b48350
I20250625 20:01:32.526037 3380 ts_tablet_manager.cc:1918] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:32.526249 3378 tablet_replica.cc:331] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:32.526365 3378 raft_consensus.cc:2241] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:32.526392 3374 ts_tablet_manager.cc:1939] T 6c8e829c96474e94a8de9e7e32b48350 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:32.526448 3379 tablet_bootstrap.cc:492] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:32.526542 3378 raft_consensus.cc:2270] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:32.526860 3378 ts_tablet_manager.cc:1905] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:32.526952 3374 tablet_replica.cc:331] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:32.527047 3374 raft_consensus.cc:2241] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:32.527140 3374 raft_consensus.cc:2270] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:32.527380 3379 tablet_bootstrap.cc:654] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:32.527441 3374 ts_tablet_manager.cc:1905] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:32.528074 3381 tablet_bootstrap.cc:492] T a90d15fb49c94d3e85a5ba26aab2878a P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:32.528246 3377 tablet_bootstrap.cc:492] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:32.528317 3377 ts_tablet_manager.cc:1397] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250625 20:01:32.528753 3377 raft_consensus.cc:357] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.528834 3378 ts_tablet_manager.cc:1918] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:32.528848 3377 raft_consensus.cc:383] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:32.528889 3378 log.cc:1199] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/6c8e829c96474e94a8de9e7e32b48350
I20250625 20:01:32.528877 3377 raft_consensus.cc:738] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:32.528990 3381 tablet_bootstrap.cc:654] T a90d15fb49c94d3e85a5ba26aab2878a P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:32.529033 3377 consensus_queue.cc:260] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.529114 3378 ts_tablet_manager.cc:1939] T 6c8e829c96474e94a8de9e7e32b48350 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:32.529229 3380 log.cc:1199] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/d1b4020c408349719f7533e4b6c0d1cd
I20250625 20:01:32.529453 3380 ts_tablet_manager.cc:1939] T d1b4020c408349719f7533e4b6c0d1cd P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:32.529872 3374 ts_tablet_manager.cc:1918] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:32.529928 3374 log.cc:1199] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/7b1d8f876f744c868857b56e49dda3e7
I20250625 20:01:32.530119 3380 tablet_replica.cc:331] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:32.530138 3374 ts_tablet_manager.cc:1939] T 7b1d8f876f744c868857b56e49dda3e7 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:32.530234 3380 raft_consensus.cc:2241] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:32.530332 3380 raft_consensus.cc:2270] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:32.530655 3380 ts_tablet_manager.cc:1905] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:32.532037 3377 ts_tablet_manager.cc:1428] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 20:01:32.532567 3380 ts_tablet_manager.cc:1918] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:32.532624 3380 log.cc:1199] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/6c8e829c96474e94a8de9e7e32b48350
I20250625 20:01:32.532825 3380 ts_tablet_manager.cc:1939] T 6c8e829c96474e94a8de9e7e32b48350 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:32.533015 3378 tablet_replica.cc:331] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:32.533115 3378 raft_consensus.cc:2241] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:32.533197 3378 raft_consensus.cc:2270] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:32.533421 3380 tablet_replica.cc:331] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:32.533507 3378 ts_tablet_manager.cc:1905] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:32.533512 3380 raft_consensus.cc:2241] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:32.533671 3380 raft_consensus.cc:2270] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:32.533968 3380 ts_tablet_manager.cc:1905] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:32.534996 3377 tablet_bootstrap.cc:492] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:32.535296 3378 ts_tablet_manager.cc:1918] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:32.535354 3378 log.cc:1199] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/7b1d8f876f744c868857b56e49dda3e7
I20250625 20:01:32.535558 3378 ts_tablet_manager.cc:1939] T 7b1d8f876f744c868857b56e49dda3e7 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:32.535578 3380 ts_tablet_manager.cc:1918] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:32.535626 3380 log.cc:1199] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/7b1d8f876f744c868857b56e49dda3e7
I20250625 20:01:32.535861 3380 ts_tablet_manager.cc:1939] T 7b1d8f876f744c868857b56e49dda3e7 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:32.536734 3379 tablet_bootstrap.cc:492] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:32.536782 3381 tablet_bootstrap.cc:492] T a90d15fb49c94d3e85a5ba26aab2878a P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:32.536808 3379 ts_tablet_manager.cc:1397] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250625 20:01:32.536842 3381 ts_tablet_manager.cc:1397] T a90d15fb49c94d3e85a5ba26aab2878a P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250625 20:01:32.537024 3377 tablet_bootstrap.cc:654] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:32.537237 3379 raft_consensus.cc:357] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.537250 3381 raft_consensus.cc:357] T a90d15fb49c94d3e85a5ba26aab2878a P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.537334 3379 raft_consensus.cc:383] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:32.537334 3381 raft_consensus.cc:383] T a90d15fb49c94d3e85a5ba26aab2878a P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:32.537365 3381 raft_consensus.cc:738] T a90d15fb49c94d3e85a5ba26aab2878a P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:32.537365 3379 raft_consensus.cc:738] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:32.537488 3379 consensus_queue.cc:260] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.537494 3381 consensus_queue.cc:260] T a90d15fb49c94d3e85a5ba26aab2878a P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.538535 3379 ts_tablet_manager.cc:1428] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:32.538698 3379 tablet_bootstrap.cc:492] T b927d1839a3b4b2690cabac2e19f74d5 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:32.538692 3381 ts_tablet_manager.cc:1428] T a90d15fb49c94d3e85a5ba26aab2878a P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:32.539302 3381 tablet_bootstrap.cc:492] T 68a1a119b2b448afabe18cc28a474f4c P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:32.539589 3379 tablet_bootstrap.cc:654] T b927d1839a3b4b2690cabac2e19f74d5 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:32.540259 3381 tablet_bootstrap.cc:654] T 68a1a119b2b448afabe18cc28a474f4c P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:32.540581 3379 tablet_bootstrap.cc:492] T b927d1839a3b4b2690cabac2e19f74d5 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:32.540647 3379 ts_tablet_manager.cc:1397] T b927d1839a3b4b2690cabac2e19f74d5 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250625 20:01:32.541076 3379 raft_consensus.cc:357] T b927d1839a3b4b2690cabac2e19f74d5 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.541162 3379 raft_consensus.cc:383] T b927d1839a3b4b2690cabac2e19f74d5 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:32.541190 3379 raft_consensus.cc:738] T b927d1839a3b4b2690cabac2e19f74d5 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:32.541294 3379 consensus_queue.cc:260] T b927d1839a3b4b2690cabac2e19f74d5 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.541553 3379 ts_tablet_manager.cc:1428] T b927d1839a3b4b2690cabac2e19f74d5 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 20:01:32.541728 3379 tablet_bootstrap.cc:492] T 68a1a119b2b448afabe18cc28a474f4c P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:32.541715 3381 tablet_bootstrap.cc:492] T 68a1a119b2b448afabe18cc28a474f4c P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:32.542260 3381 ts_tablet_manager.cc:1397] T 68a1a119b2b448afabe18cc28a474f4c P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 20:01:32.542703 3379 tablet_bootstrap.cc:654] T 68a1a119b2b448afabe18cc28a474f4c P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:32.542770 3381 raft_consensus.cc:357] T 68a1a119b2b448afabe18cc28a474f4c P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.542865 3381 raft_consensus.cc:383] T 68a1a119b2b448afabe18cc28a474f4c P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:32.542893 3381 raft_consensus.cc:738] T 68a1a119b2b448afabe18cc28a474f4c P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:32.543018 3381 consensus_queue.cc:260] T 68a1a119b2b448afabe18cc28a474f4c P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.543931 3379 tablet_bootstrap.cc:492] T 68a1a119b2b448afabe18cc28a474f4c P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:32.543998 3379 ts_tablet_manager.cc:1397] T 68a1a119b2b448afabe18cc28a474f4c P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20250625 20:01:32.544003 3381 ts_tablet_manager.cc:1428] T 68a1a119b2b448afabe18cc28a474f4c P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:32.544152 3381 tablet_bootstrap.cc:492] T b927d1839a3b4b2690cabac2e19f74d5 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:32.544377 3379 raft_consensus.cc:357] T 68a1a119b2b448afabe18cc28a474f4c P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.544463 3379 raft_consensus.cc:383] T 68a1a119b2b448afabe18cc28a474f4c P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:32.544492 3379 raft_consensus.cc:738] T 68a1a119b2b448afabe18cc28a474f4c P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:32.544606 3379 consensus_queue.cc:260] T 68a1a119b2b448afabe18cc28a474f4c P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.544934 3379 ts_tablet_manager.cc:1428] T 68a1a119b2b448afabe18cc28a474f4c P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:32.545928 3381 tablet_bootstrap.cc:654] T b927d1839a3b4b2690cabac2e19f74d5 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:32.547029 3377 tablet_bootstrap.cc:492] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:32.547114 3377 ts_tablet_manager.cc:1397] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20250625 20:01:32.547189 3381 tablet_bootstrap.cc:492] T b927d1839a3b4b2690cabac2e19f74d5 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:32.547255 3381 ts_tablet_manager.cc:1397] T b927d1839a3b4b2690cabac2e19f74d5 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 20:01:32.547600 3377 raft_consensus.cc:357] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.547708 3377 raft_consensus.cc:383] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:32.547659 3381 raft_consensus.cc:357] T b927d1839a3b4b2690cabac2e19f74d5 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.547801 3377 raft_consensus.cc:738] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:32.547837 3381 raft_consensus.cc:383] T b927d1839a3b4b2690cabac2e19f74d5 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:32.547868 3381 raft_consensus.cc:738] T b927d1839a3b4b2690cabac2e19f74d5 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:32.547943 3377 consensus_queue.cc:260] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.547972 3381 consensus_queue.cc:260] T b927d1839a3b4b2690cabac2e19f74d5 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.548200 3381 ts_tablet_manager.cc:1428] T b927d1839a3b4b2690cabac2e19f74d5 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:32.549525 3377 ts_tablet_manager.cc:1428] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 20:01:32.549732 3377 tablet_bootstrap.cc:492] T a90d15fb49c94d3e85a5ba26aab2878a P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:32.550508 2897 raft_consensus.cc:491] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:32.550587 2897 raft_consensus.cc:513] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.550913 2897 leader_election.cc:290] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:32.551054 3377 tablet_bootstrap.cc:654] T a90d15fb49c94d3e85a5ba26aab2878a P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:32.551241 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68a1a119b2b448afabe18cc28a474f4c" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:32.551347 1560 raft_consensus.cc:2466] T 68a1a119b2b448afabe18cc28a474f4c P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:32.551636 1634 leader_election.cc:304] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:32.551877 1423 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68a1a119b2b448afabe18cc28a474f4c" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:32.552003 1423 raft_consensus.cc:2466] T 68a1a119b2b448afabe18cc28a474f4c P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:32.552292 2897 raft_consensus.cc:2802] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:32.552345 2897 raft_consensus.cc:491] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:32.552378 2897 raft_consensus.cc:3058] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:32.553411 2897 raft_consensus.cc:513] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.553741 2897 leader_election.cc:290] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:32.554054 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68a1a119b2b448afabe18cc28a474f4c" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:32.554158 1560 raft_consensus.cc:3058] T 68a1a119b2b448afabe18cc28a474f4c P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:32.554312 1423 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68a1a119b2b448afabe18cc28a474f4c" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:32.554416 1423 raft_consensus.cc:3058] T 68a1a119b2b448afabe18cc28a474f4c P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:32.555102 1560 raft_consensus.cc:2466] T 68a1a119b2b448afabe18cc28a474f4c P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:32.555258 3377 tablet_bootstrap.cc:492] T a90d15fb49c94d3e85a5ba26aab2878a P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:32.555334 3377 ts_tablet_manager.cc:1397] T a90d15fb49c94d3e85a5ba26aab2878a P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20250625 20:01:32.555409 1634 leader_election.cc:304] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:32.555441 1423 raft_consensus.cc:2466] T 68a1a119b2b448afabe18cc28a474f4c P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:32.555806 3377 raft_consensus.cc:357] T a90d15fb49c94d3e85a5ba26aab2878a P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.555892 3377 raft_consensus.cc:383] T a90d15fb49c94d3e85a5ba26aab2878a P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:32.555914 3377 raft_consensus.cc:738] T a90d15fb49c94d3e85a5ba26aab2878a P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:32.556012 3377 consensus_queue.cc:260] T a90d15fb49c94d3e85a5ba26aab2878a P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.556202 2897 raft_consensus.cc:2802] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:32.556284 3377 ts_tablet_manager.cc:1428] T a90d15fb49c94d3e85a5ba26aab2878a P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 20:01:32.556376 2897 raft_consensus.cc:695] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
I20250625 20:01:32.556494 2897 consensus_queue.cc:237] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.567884 3335 raft_consensus.cc:491] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:32.568006 3335 raft_consensus.cc:513] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.568463 3335 leader_election.cc:290] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:32.568814 1560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a90d15fb49c94d3e85a5ba26aab2878a" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:32.568938 1560 raft_consensus.cc:2466] T a90d15fb49c94d3e85a5ba26aab2878a P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:32.568924 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a90d15fb49c94d3e85a5ba26aab2878a" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:32.569041 1701 raft_consensus.cc:2466] T a90d15fb49c94d3e85a5ba26aab2878a P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:32.569244 1357 leader_election.cc:304] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:32.569384 3335 raft_consensus.cc:2802] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:32.569429 3335 raft_consensus.cc:491] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:32.569459 3335 raft_consensus.cc:3058] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:32.570197 3335 raft_consensus.cc:513] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.570508 3335 leader_election.cc:290] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:32.570863 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a90d15fb49c94d3e85a5ba26aab2878a" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:32.570889 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a90d15fb49c94d3e85a5ba26aab2878a" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:32.570947 1561 raft_consensus.cc:3058] T a90d15fb49c94d3e85a5ba26aab2878a P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:32.570986 1701 raft_consensus.cc:3058] T a90d15fb49c94d3e85a5ba26aab2878a P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:32.571573 1561 raft_consensus.cc:2466] T a90d15fb49c94d3e85a5ba26aab2878a P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:32.571904 1357 leader_election.cc:304] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:32.571943 1701 raft_consensus.cc:2466] T a90d15fb49c94d3e85a5ba26aab2878a P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:32.572204 3335 raft_consensus.cc:2802] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:32.572394 3335 raft_consensus.cc:695] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:32.572527 3335 consensus_queue.cc:237] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.684868 3394 raft_consensus.cc:491] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:32.685009 3394 raft_consensus.cc:513] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.685449 3394 leader_election.cc:290] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:32.685895 1423 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b927d1839a3b4b2690cabac2e19f74d5" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:32.685909 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b927d1839a3b4b2690cabac2e19f74d5" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:32.686048 1423 raft_consensus.cc:2466] T b927d1839a3b4b2690cabac2e19f74d5 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:32.686066 1561 raft_consensus.cc:2466] T b927d1839a3b4b2690cabac2e19f74d5 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:32.686362 1634 leader_election.cc:304] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:32.686545 3394 raft_consensus.cc:2802] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:32.686600 3394 raft_consensus.cc:491] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:32.686636 3394 raft_consensus.cc:3058] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:32.687455 3394 raft_consensus.cc:513] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:32.687811 3394 leader_election.cc:290] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:32.688158 1423 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b927d1839a3b4b2690cabac2e19f74d5" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:32.688180 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b927d1839a3b4b2690cabac2e19f74d5" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:32.688259 1423 raft_consensus.cc:3058] T b927d1839a3b4b2690cabac2e19f74d5 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:32.688285 1561 raft_consensus.cc:3058] T b927d1839a3b4b2690cabac2e19f74d5 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:32.689008 1423 raft_consensus.cc:2466] T b927d1839a3b4b2690cabac2e19f74d5 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:32.689247 1561 raft_consensus.cc:2466] T b927d1839a3b4b2690cabac2e19f74d5 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:32.689266 1637 leader_election.cc:304] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:32.689414 3394 raft_consensus.cc:2802] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:32.689493 3394 raft_consensus.cc:695] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
I20250625 20:01:32.689747 3394 consensus_queue.cc:237] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
W20250625 20:01:32.698210 1635 connection.cc:537] server connection from 127.0.0.1:58004 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:32.698242 1360 connection.cc:537] server connection from 127.0.0.1:55088 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:32.698352 1635 connection.cc:537] client connection to 127.0.204.62:36889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:32.698386 1358 connection.cc:537] client connection to 127.0.204.62:36889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:32.698544 1495 connection.cc:537] client connection to 127.0.204.62:36889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:32.699357 3347 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 20:01:32.698894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.204.62:36889 (local address 127.0.204.1:46367)
0625 20:01:32.699077 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0625 20:01:32.699093 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0625 20:01:32.699186 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 20:01:32.699250 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":103}
W20250625 20:01:32.699676 3346 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 20:01:32.699058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.204.62:36889 (local address 127.0.204.2:33027)
0625 20:01:32.699535 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0625 20:01:32.699588 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":402}
W20250625 20:01:32.700184 1755 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.204.62:36889)
W20250625 20:01:32.700330 1755 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.204.62:36889)
W20250625 20:01:32.700672 1755 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.204.62:36889)
W20250625 20:01:32.700884 1755 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.204.62:36889)
W20250625 20:01:32.701360 1496 connection.cc:537] server connection from 127.0.0.1:60394 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 20:01:32.936143 3394 consensus_queue.cc:1035] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:32.940313 3394 consensus_queue.cc:1035] T 68a1a119b2b448afabe18cc28a474f4c P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:33.008005 3415 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:33.008128 3415 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:33.008558 3415 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
W20250625 20:01:33.009735 3255 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:33.009824 3254 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005): Network error: Client connection negotiation failed: client connection to 127.0.204.61:38005: connect: Connection refused (error 111)
I20250625 20:01:33.009902 3254 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9; no voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:33.010057 3415 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250625 20:01:33.027776 3335 consensus_queue.cc:1035] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:33.031677 3420 consensus_queue.cc:1035] T a90d15fb49c94d3e85a5ba26aab2878a P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:33.142944 3394 consensus_queue.cc:1035] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:33.146193 2897 consensus_queue.cc:1035] T b927d1839a3b4b2690cabac2e19f74d5 P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:33.410334 2897 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:33.410465 2897 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:33.410935 2897 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:33.412014 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.61:38005
--webserver_interface=127.0.204.61
--webserver_port=44189
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 20:01:33.412036 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:01:33.412161 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:01:33.412214 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:01:33.412386 2897 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 20:01:33.518673 3425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:01:33.518899 3425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:33.518944 3425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:01:33.522348 3425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:01:33.522418 3425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:33.522446 3425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:01:33.522468 3425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:01:33.526552 3425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.61:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.204.61
--webserver_port=44189
--never_fsync=true
--heap_profile_path=/tmp/kudu.3425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:33.527658 3425 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:33.528944 3425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:01:33.536279 3430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:33.536269 3433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:33.536273 3431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:33.536626 3425 server_base.cc:1048] running on GCE node
I20250625 20:01:33.537209 3425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:33.538193 3425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:33.539352 3425 hybrid_clock.cc:648] HybridClock initialized: now 1750881693539339 us; error 27 us; skew 500 ppm
I20250625 20:01:33.541392 3425 webserver.cc:469] Webserver started at http://127.0.204.61:44189/ using document root <none> and password file <none>
I20250625 20:01:33.541955 3425 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:33.542016 3425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:33.545567 3425 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250625 20:01:33.547749 3439 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:33.548967 3425 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250625 20:01:33.549088 3425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
uuid: "f87f9e18e5b948a4a3fb3666f106abd0"
format_stamp: "Formatted at 2025-06-25 20:01:04 on dist-test-slave-lc57"
I20250625 20:01:33.549506 3425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:33.570772 3425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:33.571498 3425 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:33.571702 3425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:33.579535 3425 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.61:38005
I20250625 20:01:33.579667 3491 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.61:38005 every 8 connection(s)
I20250625 20:01:33.581015 3425 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-1/data/info.pb
I20250625 20:01:33.584956 3492 sys_catalog.cc:263] Verifying existing consensus state
I20250625 20:01:33.588161 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 3425
I20250625 20:01:33.588899 3492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Bootstrap starting.
I20250625 20:01:33.597385 1472 heartbeater.cc:344] Connected to a master server at 127.0.204.61:38005
I20250625 20:01:33.601759 3492 log.cc:826] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:33.653499 1609 heartbeater.cc:344] Connected to a master server at 127.0.204.61:38005
I20250625 20:01:33.655215 3492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=109 overwritten=0 applied=109 ignored=0} inserts{seen=72 ignored=0} mutations{seen=98 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:01:33.655592 3492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: Bootstrap complete.
I20250625 20:01:33.658700 3492 raft_consensus.cc:357] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:33.659026 3492 raft_consensus.cc:738] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87f9e18e5b948a4a3fb3666f106abd0, State: Initialized, Role: FOLLOWER
I20250625 20:01:33.659480 3492 consensus_queue.cc:260] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 109, Last appended: 8.109, Last appended by leader: 109, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:33.659934 3502 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:33.660054 3502 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:33.660578 3492 sys_catalog.cc:564] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:01:33.666782 3516 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:33.666858 3516 catalog_manager.cc:1554] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: loading cluster ID for follower catalog manager: success
I20250625 20:01:33.669718 3516 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: acquiring CA information for follower catalog manager: success
I20250625 20:01:33.668359 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:01:33.670411 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:33.671855 3516 catalog_manager.cc:1604] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 20:01:33.705770 3520 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:33.705909 3520 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:33.706693 3520 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
W20250625 20:01:33.708128 3255 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:33.711151 3466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" candidate_term: 10 candidate_status { last_received { term: 9 index: 115 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" is_pre_election: true
I20250625 20:01:33.711483 3466 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fcdee0c0aac4013a5e3c77304b474d9 in term 8.
I20250625 20:01:33.711833 3254 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0; no voters: 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:33.712059 3520 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250625 20:01:33.712106 3520 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:33.712136 3520 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 9 FOLLOWER]: Advancing to term 10
I20250625 20:01:33.712958 3520 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:33.713282 3520 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 10 election: Requested vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005)
I20250625 20:01:33.713588 3466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" candidate_term: 10 candidate_status { last_received { term: 9 index: 115 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0"
I20250625 20:01:33.713701 3466 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 8 FOLLOWER]: Advancing to term 10
I20250625 20:01:33.715476 3466 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcdee0c0aac4013a5e3c77304b474d9 in term 10.
I20250625 20:01:33.715778 3254 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0; no voters:
I20250625 20:01:33.715967 3520 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 10 FOLLOWER]: Leader election won for term 10
I20250625 20:01:33.716176 3520 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [term 10 LEADER]: Becoming Leader. State: Replica: 1fcdee0c0aac4013a5e3c77304b474d9, State: Running, Role: LEADER
W20250625 20:01:33.716279 3255 leader_election.cc:336] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:33.716487 3520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 9.115, Last appended by leader: 115, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:33.718127 3524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fcdee0c0aac4013a5e3c77304b474d9. Latest consensus state: current_term: 10 leader_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:33.718243 3524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9 [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:33.718545 3526 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:01:33.720360 3526 catalog_manager.cc:671] Loaded metadata for table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]
I20250625 20:01:33.720804 3526 catalog_manager.cc:671] Loaded metadata for table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]
I20250625 20:01:33.721005 3526 catalog_manager.cc:671] Loaded metadata for table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]
I20250625 20:01:33.721210 3526 catalog_manager.cc:671] Loaded metadata for table default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4]
I20250625 20:01:33.721405 3526 catalog_manager.cc:671] Loaded metadata for table default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]
I20250625 20:01:33.721611 3526 catalog_manager.cc:671] Loaded metadata for table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]
I20250625 20:01:33.721848 3526 catalog_manager.cc:671] Loaded metadata for table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996]
I20250625 20:01:33.722069 3526 catalog_manager.cc:671] Loaded metadata for table default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5]
I20250625 20:01:33.722272 3526 catalog_manager.cc:671] Loaded metadata for table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583]
I20250625 20:01:33.722427 3526 catalog_manager.cc:671] Loaded metadata for table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277]
I20250625 20:01:33.722743 3526 catalog_manager.cc:671] Loaded metadata for table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]
I20250625 20:01:33.722935 3526 catalog_manager.cc:671] Loaded metadata for table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f]
I20250625 20:01:33.723142 3526 catalog_manager.cc:671] Loaded metadata for table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]
I20250625 20:01:33.723335 3526 catalog_manager.cc:671] Loaded metadata for table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]
I20250625 20:01:33.723522 3526 catalog_manager.cc:671] Loaded metadata for table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]
I20250625 20:01:33.724988 3526 tablet_loader.cc:96] loaded metadata for tablet 01f62ece8947463daef2b44571d4dec0 (table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e])
I20250625 20:01:33.725188 3526 tablet_loader.cc:96] loaded metadata for tablet 06ef5241d21449169f228cfa37ff91b4 (table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996])
I20250625 20:01:33.725811 3526 tablet_loader.cc:96] loaded metadata for tablet 3dd7cedb1c3542c086ebae940840147b (table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8])
I20250625 20:01:33.726202 3526 tablet_loader.cc:96] loaded metadata for tablet 68a1a119b2b448afabe18cc28a474f4c (table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35])
I20250625 20:01:33.726349 3526 tablet_loader.cc:96] loaded metadata for tablet 6b660df5bca642af848d7464859ac7fc (table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa])
I20250625 20:01:33.726737 3526 tablet_loader.cc:96] loaded metadata for tablet 78631bcca66d4b16b344e57f8a1cc291 (table default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5])
I20250625 20:01:33.727036 3526 tablet_loader.cc:96] loaded metadata for tablet 912f25c1234a4f3199bd22cb33ea3788 (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809])
I20250625 20:01:33.727417 3526 tablet_loader.cc:96] loaded metadata for tablet a90d15fb49c94d3e85a5ba26aab2878a (table default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4])
I20250625 20:01:33.727823 3526 tablet_loader.cc:96] loaded metadata for tablet b927d1839a3b4b2690cabac2e19f74d5 (table default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7])
I20250625 20:01:33.728067 3526 tablet_loader.cc:96] loaded metadata for tablet bb6c755fce6640d2a0b13a449d3adfb9 (table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583])
I20250625 20:01:33.728219 3526 tablet_loader.cc:96] loaded metadata for tablet c00e8f8653bd42fda7bef627e99930cc (table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f])
I20250625 20:01:33.728521 3526 tablet_loader.cc:96] loaded metadata for tablet cdb2a543898642d3abc3afa850178120 (table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277])
I20250625 20:01:33.728972 3526 tablet_loader.cc:96] loaded metadata for tablet d8dd4c0e80974a67b63293d83760fd8b (table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36])
I20250625 20:01:33.729275 3526 tablet_loader.cc:96] loaded metadata for tablet e44908760f6245a282f72cec3b5773fe (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0])
I20250625 20:01:33.729416 3526 tablet_loader.cc:96] loaded metadata for tablet eb717c706f3d428bbd5bf6c8e4f839aa (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1])
I20250625 20:01:33.729547 3526 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:01:33.730333 3526 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:33.730378 3526 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:01:33.731240 3526 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:01:33.731971 3526 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1fcdee0c0aac4013a5e3c77304b474d9: Loaded TSK: 0
I20250625 20:01:33.732677 3526 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:01:33.733387 3526 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 20:01:33.734072 3526 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 21
I20250625 20:01:33.757519 3269 master_service.cc:948] ReplaceTablet: received request to replace tablet d1b4020c408349719f7533e4b6c0d1cd from {username='slave'} at 127.0.0.1:55190
I20250625 20:01:33.758325 3263 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55190:
name: "default.Table_a31a093a024346aab31bc8d55a00253d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:33.758414 3270 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55190:
name: "default.Table_c95f91afca114793990044d26cd96c64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:33.758771 3267 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55190:
name: "default.Table_ac867d9f2ac54f09a848a71cd7a01d49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:33.758919 3266 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55190:
name: "default.Table_3ef4196a2a204be1bb572abdbef3264c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:33.759009 3265 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55190:
name: "default.Table_d4ed5ace6c144df6af9ff3155b29c5bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:33.759140 3269 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55190:
table { table_name: "default.Table_fd28d4c5b48245c382a334b98a3066df" } modify_external_catalogs: true
I20250625 20:01:33.759213 3262 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55190:
table { table_name: "default.Table_d0f8eb84406f4d758e34b162746daa38" } new_table_name: "default.Table_a78f24eb6417409489ec4469005095b4" modify_external_catalogs: true
I20250625 20:01:33.759246 3264 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55190:
table { table_name: "default.Table_8302314b76364eadb44f02cc6d58a589" } new_table_name: "default.Table_2fcd1dbf63224757b5e1a9bad86e6a77" modify_external_catalogs: true
I20250625 20:01:33.759260 3269 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55190:
table { table_name: "default.Table_fd28d4c5b48245c382a334b98a3066df" } modify_external_catalogs: true
I20250625 20:01:33.759428 3261 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55190:
table { table_name: "default.Table_1d0eb88cbdfd4b16bb66bfb3d502672b" } new_table_name: "default.Table_dd960e4b8a9c42309150252433773c76" modify_external_catalogs: true
I20250625 20:01:33.758327 3268 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55190:
name: "default.Table_bb2207463de346c3b1a4271f21eed795"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250625 20:01:33.763273 3329 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 6 out of 13 received events: NotificationEvent(eventId=22, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_063972c2ee4b4e468cf982177f09f91a, message={"server":"","servicePrincipal":"","db":"default","table":"table_063972c2ee4b4e468cf982177f09f91a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_063972c2ee4b4e468cf982177f09f91a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_063972c2ee4b4e468cf982177f09f91a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_063972c2ee4b4e468cf982177f09f91a\",\"kudu.table_id\":\"a054c42074394f1db21dfcf2c55d99b5\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_01c13560571f419e9b6f45edee3a2a37, message={"server":"","servicePrincipal":"","db":"default","table":"table_01c13560571f419e9b6f45edee3a2a37","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_01c13560571f419e9b6f45edee3a2a37\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_01c13560571f419e9b6f45edee3a2a37\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_01c13560571f419e9b6f45edee3a2a37\",\"kudu.table_id\":\"aee290f61c8f4d7b93d4cd7192eb9277\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_a31a093a024346aab31bc8d55a00253d, message={"server":"","servicePrincipal":"","db":"default","table":"table_a31a093a024346aab31bc8d55a00253d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a31a093a024346aab31bc8d55a00253d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_a31a093a024346aab31bc8d55a00253d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a31a093a024346aab31bc8d55a00253d\",\"kudu.table_id\":\"81aee385d1404c82aaeebee220cb9e68\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_60554f72b1aa48aa93aaea511a985fe7, message={"server":"","servicePrincipal":"","db":"default","table":"table_60554f72b1aa48aa93aaea511a985fe7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_60554f72b1aa48aa93aaea511a985fe7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881667},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_60554f72b1aa48aa93aaea511a985fe7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_60554f72b1aa48aa93aaea511a985fe7\",\"kudu.table_id\":\"e46f78618ca648e19c3935d0bf874a35\",\"transient_lastDdlTime\":\"1750881667\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fe1e4529dcd google::LogMessage::Fail() at ??:0
@ 0x7fe1e452db93 google::LogMessage::SendToLog() at ??:0
@ 0x7fe1e45297cc google::LogMessage::Flush() at ??:0
@ 0x7fe1e4529a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fe1e94c2c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fe1e94c1ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fe1e94c130b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
@ 0x7fe1e94c6a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5610c6e8a752 std::function<>::operator()() at ??:0
@ 0x7fe1e597b0a1 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe1e616a6db start_thread at ??:0
@ 0x7fe1e36e071f clone at ??:0
W20250625 20:01:33.941982 1757 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.204.60:46191)
W20250625 20:01:33.942268 1757 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.204.60:46191)
W20250625 20:01:33.942430 1757 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.204.60:46191)
W20250625 20:01:33.942505 1757 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.204.60:46191)
I20250625 20:01:34.083391 1749 heartbeater.cc:344] Connected to a master server at 127.0.204.61:38005
I20250625 20:01:34.084954 3456 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" instance_seqno: 1750881665979916) as {username='slave'} at 127.0.204.3:43379; Asking this server to re-register.
I20250625 20:01:34.085424 1749 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:34.086716 3456 ts_manager.cc:194] Registered new tserver with Master: 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:34.100215 1472 heartbeater.cc:344] Connected to a master server at 127.0.204.61:38005
I20250625 20:01:34.101480 3456 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" instance_seqno: 1750881665574847) as {username='slave'} at 127.0.204.1:50233; Asking this server to re-register.
I20250625 20:01:34.101934 1472 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:34.102991 3456 ts_manager.cc:194] Registered new tserver with Master: 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:34.155984 1609 heartbeater.cc:344] Connected to a master server at 127.0.204.61:38005
I20250625 20:01:34.157167 3456 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c73846c0665d4d53a274d053946107df" instance_seqno: 1750881665791183) as {username='slave'} at 127.0.204.2:51711; Asking this server to re-register.
I20250625 20:01:34.157593 1609 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:34.158495 3456 ts_manager.cc:194] Registered new tserver with Master: c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:34.459414 3541 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:34.459542 3541 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:34.460726 3541 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:34.461428 3443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:01:34.461437 3441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:34.462388 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:34.462497 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:34.462560 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:34.462777 3541 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250625 20:01:35.137085 3547 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:35.137212 3547 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:35.137687 3547 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:35.139122 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:35.139235 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:35.139300 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:35.139436 3547 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250625 20:01:36.261966 3555 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:36.262099 3555 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:36.262537 3555 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:36.263746 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:01:36.263823 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:36.263938 3441 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:36.264081 3555 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250625 20:01:36.753295 816 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
/tmp/dist-test-tasknmPaw0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.62:36889
--webserver_interface=127.0.204.62
--webserver_port=37479
--builtin_ntp_servers=127.0.204.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 20:01:36.856514 3562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:01:36.856726 3562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:01:36.856762 3562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:01:36.859866 3562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:01:36.859922 3562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:01:36.859941 3562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:01:36.859956 3562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:01:36.863612 3562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.204.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40079
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.204.62:36889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.204.62
--webserver_port=37479
--never_fsync=true
--heap_profile_path=/tmp/kudu.3562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 25 Jun 2025 19:43:17 UTC on 24a791456cd2
build id 6766
I20250625 20:01:36.864718 3562 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:01:36.865754 3562 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250625 20:01:36.869701 3571 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:36.869828 3571 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:36.870281 3571 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:36.871631 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:36.871764 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:36.871877 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:36.872043 3571 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250625 20:01:36.872359 3568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:36.872366 3570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:01:36.872360 3567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:01:36.872700 3562 server_base.cc:1048] running on GCE node
I20250625 20:01:36.873174 3562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:01:36.874186 3562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:01:36.875349 3562 hybrid_clock.cc:648] HybridClock initialized: now 1750881696875330 us; error 30 us; skew 500 ppm
I20250625 20:01:36.882249 3562 webserver.cc:469] Webserver started at http://127.0.204.62:37479/ using document root <none> and password file <none>
I20250625 20:01:36.882776 3562 fs_manager.cc:362] Metadata directory not provided
I20250625 20:01:36.882825 3562 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:01:36.886191 3562 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:36.888244 3580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:01:36.889302 3562 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250625 20:01:36.889391 3562 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data,/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
uuid: "5852fb53503b438c92f294e0c44c916d"
format_stamp: "Formatted at 2025-06-25 20:01:04 on dist-test-slave-lc57"
I20250625 20:01:36.889765 3562 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:01:36.900745 3562 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:01:36.901409 3562 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:01:36.901587 3562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:01:36.908593 3632 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.204.62:36889 every 8 connection(s)
I20250625 20:01:36.908589 3562 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.62:36889
I20250625 20:01:36.909746 3562 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/master-0/data/info.pb
I20250625 20:01:36.913555 3633 sys_catalog.cc:263] Verifying existing consensus state
I20250625 20:01:36.915102 3633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Bootstrap starting.
I20250625 20:01:36.919621 816 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasknmPaw0/build/debug/bin/kudu as pid 3562
I20250625 20:01:36.927796 3633 log.cc:826] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Log is configured to *not* fsync() on all Append() calls
I20250625 20:01:36.982671 3633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Bootstrap replayed 1/1 log segments. Stats: ops{read=116 overwritten=1 applied=115 ignored=0} inserts{seen=75 ignored=0} mutations{seen=104 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:01:36.983073 3633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Bootstrap complete.
I20250625 20:01:36.985899 3633 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:36.986310 3633 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5852fb53503b438c92f294e0c44c916d, State: Initialized, Role: FOLLOWER
I20250625 20:01:36.986827 3633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 115, Last appended: 9.115, Last appended by leader: 115, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:36.987303 3637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:36.987421 3637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:36.988046 3633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:01:36.995743 3651 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:36.995893 3651 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: loading cluster ID for follower catalog manager: success
I20250625 20:01:37.001297 3651 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: acquiring CA information for follower catalog manager: success
I20250625 20:01:37.003741 3651 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 20:01:37.010435 3562 master_runner.cc:186] Error getting master registration for 127.0.204.60:46191: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111),
I20250625 20:01:37.012344 3562 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:37.090023 1748 heartbeater.cc:344] Connected to a master server at 127.0.204.62:36889
I20250625 20:01:37.092052 3597 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" instance_seqno: 1750881665979916) as {username='slave'} at 127.0.204.3:34535; Asking this server to re-register.
I20250625 20:01:37.093779 1748 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:37.095589 3597 ts_manager.cc:194] Registered new tserver with Master: 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:37.096526 1471 heartbeater.cc:344] Connected to a master server at 127.0.204.62:36889
I20250625 20:01:37.097671 3597 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" instance_seqno: 1750881665574847) as {username='slave'} at 127.0.204.1:53113; Asking this server to re-register.
I20250625 20:01:37.098093 1471 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:37.099035 3597 ts_manager.cc:194] Registered new tserver with Master: 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759)
I20250625 20:01:37.158667 1608 heartbeater.cc:344] Connected to a master server at 127.0.204.62:36889
I20250625 20:01:37.159827 3597 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c73846c0665d4d53a274d053946107df" instance_seqno: 1750881665791183) as {username='slave'} at 127.0.204.2:32913; Asking this server to re-register.
I20250625 20:01:37.160250 1608 heartbeater.cc:461] Registering TS with master...
I20250625 20:01:37.161176 3597 ts_manager.cc:194] Registered new tserver with Master: c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:37.392333 3637 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:37.392544 3637 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:37.393689 3637 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:37.394106 3466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 10 candidate_status { last_received { term: 9 index: 115 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" is_pre_election: true
W20250625 20:01:37.394150 3584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:37.394311 3466 raft_consensus.cc:2391] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5852fb53503b438c92f294e0c44c916d in current term 10: Already voted for candidate 1fcdee0c0aac4013a5e3c77304b474d9 in this term.
W20250625 20:01:37.394913 3584 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:37.394981 3584 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, f87f9e18e5b948a4a3fb3666f106abd0
I20250625 20:01:37.395179 3637 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 9 FOLLOWER]: Advancing to term 10
I20250625 20:01:37.397114 3637 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250625 20:01:37.999825 3660 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:37.999946 3660 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:38.000339 3660 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:38.000768 3466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 11 candidate_status { last_received { term: 9 index: 115 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" is_pre_election: true
I20250625 20:01:38.000913 3466 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5852fb53503b438c92f294e0c44c916d in term 10.
I20250625 20:01:38.001204 3581 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0; no voters:
I20250625 20:01:38.001382 3660 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250625 20:01:38.001458 3660 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:38.001506 3660 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 10 FOLLOWER]: Advancing to term 11
W20250625 20:01:38.001570 3584 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:38.002403 3660 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:38.002707 3660 leader_election.cc:290] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 11 election: Requested vote from peers f87f9e18e5b948a4a3fb3666f106abd0 (127.0.204.61:38005), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
I20250625 20:01:38.003067 3466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5852fb53503b438c92f294e0c44c916d" candidate_term: 11 candidate_status { last_received { term: 9 index: 115 } } ignore_live_leader: false dest_uuid: "f87f9e18e5b948a4a3fb3666f106abd0"
I20250625 20:01:38.003201 3466 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 10 FOLLOWER]: Advancing to term 11
W20250625 20:01:38.003624 3584 leader_election.cc:336] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:38.004184 3466 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5852fb53503b438c92f294e0c44c916d in term 11.
I20250625 20:01:38.004465 3581 leader_election.cc:304] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5852fb53503b438c92f294e0c44c916d, f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9
I20250625 20:01:38.004652 3660 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 11 FOLLOWER]: Leader election won for term 11
I20250625 20:01:38.004802 3660 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [term 11 LEADER]: Becoming Leader. State: Replica: 5852fb53503b438c92f294e0c44c916d, State: Running, Role: LEADER
I20250625 20:01:38.005196 3660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 9.115, Last appended by leader: 115, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:38.006819 3663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5852fb53503b438c92f294e0c44c916d. Latest consensus state: current_term: 11 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:38.006925 3663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:38.007215 3665 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:01:38.009038 3665 catalog_manager.cc:671] Loaded metadata for table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa]
I20250625 20:01:38.009479 3665 catalog_manager.cc:671] Loaded metadata for table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1]
I20250625 20:01:38.009681 3665 catalog_manager.cc:671] Loaded metadata for table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36]
I20250625 20:01:38.009871 3665 catalog_manager.cc:671] Loaded metadata for table default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4]
I20250625 20:01:38.010061 3665 catalog_manager.cc:671] Loaded metadata for table default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7]
I20250625 20:01:38.010250 3665 catalog_manager.cc:671] Loaded metadata for table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8]
I20250625 20:01:38.010440 3665 catalog_manager.cc:671] Loaded metadata for table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996]
I20250625 20:01:38.010633 3665 catalog_manager.cc:671] Loaded metadata for table default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5]
I20250625 20:01:38.010838 3665 catalog_manager.cc:671] Loaded metadata for table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583]
I20250625 20:01:38.011034 3665 catalog_manager.cc:671] Loaded metadata for table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277]
I20250625 20:01:38.011336 3665 catalog_manager.cc:671] Loaded metadata for table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]
I20250625 20:01:38.011574 3665 catalog_manager.cc:671] Loaded metadata for table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f]
I20250625 20:01:38.011821 3665 catalog_manager.cc:671] Loaded metadata for table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35]
I20250625 20:01:38.012040 3665 catalog_manager.cc:671] Loaded metadata for table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809]
I20250625 20:01:38.012233 3665 catalog_manager.cc:671] Loaded metadata for table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]
I20250625 20:01:38.013648 3665 tablet_loader.cc:96] loaded metadata for tablet 01f62ece8947463daef2b44571d4dec0 (table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e])
I20250625 20:01:38.013810 3665 tablet_loader.cc:96] loaded metadata for tablet 06ef5241d21449169f228cfa37ff91b4 (table default.table_8302314b76364eadb44f02cc6d58a589 [id=82357871d20c400dbb0c5727da020996])
I20250625 20:01:38.014405 3665 tablet_loader.cc:96] loaded metadata for tablet 3dd7cedb1c3542c086ebae940840147b (table default.table_fd28d4c5b48245c382a334b98a3066df [id=77d32088c98540ec84ca62e7c8750cb8])
I20250625 20:01:38.014763 3665 tablet_loader.cc:96] loaded metadata for tablet 68a1a119b2b448afabe18cc28a474f4c (table default.table_60554f72b1aa48aa93aaea511a985fe7 [id=e46f78618ca648e19c3935d0bf874a35])
I20250625 20:01:38.014884 3665 tablet_loader.cc:96] loaded metadata for tablet 6b660df5bca642af848d7464859ac7fc (table default.table_312eaaab0028455bb53f1af2373b85ab [id=0f4987c9c14c46a89443ce1b66ceaaaa])
I20250625 20:01:38.015333 3665 tablet_loader.cc:96] loaded metadata for tablet 78631bcca66d4b16b344e57f8a1cc291 (table default.table_063972c2ee4b4e468cf982177f09f91a [id=a054c42074394f1db21dfcf2c55d99b5])
I20250625 20:01:38.015689 3665 tablet_loader.cc:96] loaded metadata for tablet 912f25c1234a4f3199bd22cb33ea3788 (table default.table_030bc68e40784cf2a6bd261221a0f738 [id=e6ca598bb22a45abb67ce2e547f3a809])
I20250625 20:01:38.016207 3665 tablet_loader.cc:96] loaded metadata for tablet a90d15fb49c94d3e85a5ba26aab2878a (table default.table_c1154a92402c4c0485b49679f754ecb4 [id=66512a85e5d34065afe0c5ca62599bc4])
I20250625 20:01:38.016634 3665 tablet_loader.cc:96] loaded metadata for tablet b927d1839a3b4b2690cabac2e19f74d5 (table default.table_efbb957abb8f444ea073162901fa6215 [id=71733c4c432244469f60c9ffeb917cc7])
I20250625 20:01:38.016841 3665 tablet_loader.cc:96] loaded metadata for tablet bb6c755fce6640d2a0b13a449d3adfb9 (table default.table_976af21482a74bc79eb800684cce978e [id=a8f816328d6c43b8b796227333a1d583])
I20250625 20:01:38.016961 3665 tablet_loader.cc:96] loaded metadata for tablet c00e8f8653bd42fda7bef627e99930cc (table default.table_03571335de5b4e4fb1fb01b5e6f4a6fb [id=de00fa1efd2041b29fa3e9d329f6043f])
I20250625 20:01:38.017227 3665 tablet_loader.cc:96] loaded metadata for tablet cdb2a543898642d3abc3afa850178120 (table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277])
I20250625 20:01:38.017650 3665 tablet_loader.cc:96] loaded metadata for tablet d8dd4c0e80974a67b63293d83760fd8b (table default.table_ef511cc05ad8422e98275bbb9b82ecb4 [id=5483979a624546ea93cff095a5cd2a36])
I20250625 20:01:38.017915 3665 tablet_loader.cc:96] loaded metadata for tablet e44908760f6245a282f72cec3b5773fe (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0])
I20250625 20:01:38.018033 3665 tablet_loader.cc:96] loaded metadata for tablet eb717c706f3d428bbd5bf6c8e4f839aa (table default.table_d0f8eb84406f4d758e34b162746daa38 [id=4e3fbfc6b591464886cb1e94527676c1])
I20250625 20:01:38.018146 3665 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:01:38.018880 3665 catalog_manager.cc:1261] Loaded cluster ID: 78906ff060484834878da5d4bb8fbd38
I20250625 20:01:38.018925 3665 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:01:38.019783 3665 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:01:38.020432 3665 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Loaded TSK: 0
I20250625 20:01:38.021116 3665 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:01:38.021770 3665 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 20:01:38.022372 3665 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 21
I20250625 20:01:38.069794 3597 master_service.cc:948] ReplaceTablet: received request to replace tablet cdb2a543898642d3abc3afa850178120 from {username='slave'} at 127.0.0.1:37162
I20250625 20:01:38.069904 3596 master_service.cc:948] ReplaceTablet: received request to replace tablet e44908760f6245a282f72cec3b5773fe from {username='slave'} at 127.0.0.1:37162
I20250625 20:01:38.070196 3595 master_service.cc:948] ReplaceTablet: received request to replace tablet 01f62ece8947463daef2b44571d4dec0 from {username='slave'} at 127.0.0.1:37162
I20250625 20:01:38.070254 3592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37162:
table { table_name: "default.Table_ef511cc05ad8422e98275bbb9b82ecb4" } modify_external_catalogs: true
I20250625 20:01:38.070395 3592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37162:
table { table_name: "default.Table_ef511cc05ad8422e98275bbb9b82ecb4" } modify_external_catalogs: true
I20250625 20:01:38.070734 3591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37162:
table { table_name: "default.Table_063972c2ee4b4e468cf982177f09f91a" } new_table_name: "default.Table_980a5f7891e24ccc87c72a858cc5076f" modify_external_catalogs: true
I20250625 20:01:38.071099 3588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37162:
name: "default.Table_d4ed5ace6c144df6af9ff3155b29c5bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:38.071199 3590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37162:
name: "default.Table_3629c083428e4ae0a977ffb16af83bc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:38.071079 3589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37162:
table { table_name: "default.Table_01c13560571f419e9b6f45edee3a2a37" } modify_external_catalogs: true
I20250625 20:01:38.071483 3589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37162:
table { table_name: "default.Table_01c13560571f419e9b6f45edee3a2a37" } modify_external_catalogs: true
I20250625 20:01:38.072151 3594 master_service.cc:948] ReplaceTablet: received request to replace tablet e44908760f6245a282f72cec3b5773fe from {username='slave'} at 127.0.0.1:37162
I20250625 20:01:38.072167 3593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37162:
table { table_name: "default.Table_030bc68e40784cf2a6bd261221a0f738" } modify_external_catalogs: true
I20250625 20:01:38.072273 3593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37162:
table { table_name: "default.Table_030bc68e40784cf2a6bd261221a0f738" } modify_external_catalogs: true
W20250625 20:01:38.074618 3584 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d -> Peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Couldn't send request to peer 1fcdee0c0aac4013a5e3c77304b474d9. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 20:01:38.075153 3466 raft_consensus.cc:1273] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Refusing update from remote peer 5852fb53503b438c92f294e0c44c916d: Log matching property violated. Preceding OpId in replica: term: 8 index: 109. Preceding OpId from leader: term: 11 index: 117. (index mismatch)
I20250625 20:01:38.075692 3663 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 109, Time since last communication: 0.000s
F20250625 20:01:38.076956 3650 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 6 out of 13 received events: NotificationEvent(eventId=22, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_063972c2ee4b4e468cf982177f09f91a, message={"server":"","servicePrincipal":"","db":"default","table":"table_063972c2ee4b4e468cf982177f09f91a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_063972c2ee4b4e468cf982177f09f91a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_063972c2ee4b4e468cf982177f09f91a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_063972c2ee4b4e468cf982177f09f91a\",\"kudu.table_id\":\"a054c42074394f1db21dfcf2c55d99b5\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1750881668, eventType=CREATE_TABLE, dbName=default, tableName=table_01c13560571f419e9b6f45edee3a2a37, message={"server":"","servicePrincipal":"","db":"default","table":"table_01c13560571f419e9b6f45edee3a2a37","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_01c13560571f419e9b6f45edee3a2a37\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_01c13560571f419e9b6f45edee3a2a37\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_01c13560571f419e9b6f45edee3a2a37\",\"kudu.table_id\":\"aee290f61c8f4d7b93d4cd7192eb9277\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881668,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_a31a093a024346aab31bc8d55a00253d, message={"server":"","servicePrincipal":"","db":"default","table":"table_a31a093a024346aab31bc8d55a00253d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a31a093a024346aab31bc8d55a00253d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_a31a093a024346aab31bc8d55a00253d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a31a093a024346aab31bc8d55a00253d\",\"kudu.table_id\":\"81aee385d1404c82aaeebee220cb9e68\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_d4ed5ace6c144df6af9ff3155b29c5bb, message={"server":"","servicePrincipal":"","db":"default","table":"table_d4ed5ace6c144df6af9ff3155b29c5bb","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_d4ed5ace6c144df6af9ff3155b29c5bb\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d4ed5ace6c144df6af9ff3155b29c5bb\",\"kudu.table_id\":\"65222a2a9f7e44099a490760d1f51ce4\",\"transient_lastDdlTime\":\"1750881668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1750881669, eventType=DROP_TABLE, dbName=default, tableName=table_60554f72b1aa48aa93aaea511a985fe7, message={"server":"","servicePrincipal":"","db":"default","table":"table_60554f72b1aa48aa93aaea511a985fe7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_60554f72b1aa48aa93aaea511a985fe7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750881667},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/warehouse/table_60554f72b1aa48aa93aaea511a985fe7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_60554f72b1aa48aa93aaea511a985fe7\",\"kudu.table_id\":\"e46f78618ca648e19c3935d0bf874a35\",\"transient_lastDdlTime\":\"1750881667\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"78906ff060484834878da5d4bb8fbd38\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750881669}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f51f5471dcd google::LogMessage::Fail() at ??:0
@ 0x7f51f5475b93 google::LogMessage::SendToLog() at ??:0
I20250625 20:01:38.082768 3672 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5852fb53503b438c92f294e0c44c916d. Latest consensus state: current_term: 11 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:38.082894 3672 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:38.082974 3672 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:38.083029 3672 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f51f54717cc google::LogMessage::Flush() at ??:0
I20250625 20:01:38.085444 3660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 11 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
@ 0x7f51f5471a11 google::LogMessage::~LogMessage() at ??:0
I20250625 20:01:38.085587 3660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d [sys.catalog]: This master's current role is: LEADER
I20250625 20:01:38.085698 3667 mvcc.cc:204] Tried to move back new op lower bound from 7171611435311357952 to 7171611435031633920. Current Snapshot: MvccSnapshot[applied={T|T < 7171611435305885696}]
I20250625 20:01:38.085899 3672 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "5852fb53503b438c92f294e0c44c916d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } } }
I20250625 20:01:38.085996 3672 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 20:01:38.086793 3595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Sending DeleteTablet for 3 replicas of tablet 01f62ece8947463daef2b44571d4dec0
I20250625 20:01:38.088085 3595 catalog_manager.cc:6652] ReplaceTablet: tablet 01f62ece8947463daef2b44571d4dec0 deleted and replaced by tablet ad57e94ba6d048c19b763ef6cb4f0fcb
I20250625 20:01:38.088302 3596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Sending DeleteTablet for 3 replicas of tablet e44908760f6245a282f72cec3b5773fe
I20250625 20:01:38.088464 3595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37162:
name: "default.Table_3ef4196a2a204be1bb572abdbef3264c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:38.088905 3597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5852fb53503b438c92f294e0c44c916d: Sending DeleteTablet for 3 replicas of tablet cdb2a543898642d3abc3afa850178120
I20250625 20:01:38.089051 3594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37162:
name: "default.Table_ac867d9f2ac54f09a848a71cd7a01d49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:38.089133 3597 catalog_manager.cc:6652] ReplaceTablet: tablet cdb2a543898642d3abc3afa850178120 deleted and replaced by tablet 26ac469f2b9f49efa88bbed6826ac438
I20250625 20:01:38.089318 3597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37162:
name: "default.Table_c95f91afca114793990044d26cd96c64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:01:38.093271 3670 mvcc.cc:204] Tried to move back new op lower bound from 7171611435311357952 to 7171611412137127936. Current Snapshot: MvccSnapshot[applied={T|T < 7171611412462387200}]
@ 0x7f51fa40ac91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250625 20:01:38.088565 3596 catalog_manager.cc:6652] ReplaceTablet: tablet e44908760f6245a282f72cec3b5773fe deleted and replaced by tablet bb3e9ff4d9c7400f889052c738a8cfda
I20250625 20:01:38.094744 3596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37162:
name: "default.Table_bb2207463de346c3b1a4271f21eed795"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f51fa409ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f51fa40930b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250625 20:01:38.102954 1534 tablet_service.cc:1515] Processing DeleteTablet for tablet 01f62ece8947463daef2b44571d4dec0 with delete_type TABLET_DATA_DELETED (replaced by tablet ad57e94ba6d048c19b763ef6cb4f0fcb) from {username='slave'} at 127.0.0.1:39840
I20250625 20:01:38.103200 1538 tablet_service.cc:1515] Processing DeleteTablet for tablet e44908760f6245a282f72cec3b5773fe with delete_type TABLET_DATA_DELETED (replaced by tablet bb3e9ff4d9c7400f889052c738a8cfda) from {username='slave'} at 127.0.0.1:39840
I20250625 20:01:38.103444 1673 tablet_service.cc:1515] Processing DeleteTablet for tablet 01f62ece8947463daef2b44571d4dec0 with delete_type TABLET_DATA_DELETED (replaced by tablet ad57e94ba6d048c19b763ef6cb4f0fcb) from {username='slave'} at 127.0.0.1:43112
I20250625 20:01:38.103449 1534 tablet_service.cc:1468] Processing CreateTablet for tablet ad57e94ba6d048c19b763ef6cb4f0fcb (DEFAULT_TABLE table=default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:38.103582 1537 tablet_service.cc:1468] Processing CreateTablet for tablet 26ac469f2b9f49efa88bbed6826ac438 (DEFAULT_TABLE table=default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:38.103679 1676 tablet_service.cc:1515] Processing DeleteTablet for tablet e44908760f6245a282f72cec3b5773fe with delete_type TABLET_DATA_DELETED (replaced by tablet bb3e9ff4d9c7400f889052c738a8cfda) from {username='slave'} at 127.0.0.1:43112
I20250625 20:01:38.103737 1534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad57e94ba6d048c19b763ef6cb4f0fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:38.103816 1675 tablet_service.cc:1515] Processing DeleteTablet for tablet cdb2a543898642d3abc3afa850178120 with delete_type TABLET_DATA_DELETED (replaced by tablet 26ac469f2b9f49efa88bbed6826ac438) from {username='slave'} at 127.0.0.1:43112
I20250625 20:01:38.103852 1537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ac469f2b9f49efa88bbed6826ac438. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:38.104055 1672 tablet_service.cc:1468] Processing CreateTablet for tablet 26ac469f2b9f49efa88bbed6826ac438 (DEFAULT_TABLE table=default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:38.104328 1672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ac469f2b9f49efa88bbed6826ac438. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:38.103749 1536 tablet_service.cc:1515] Processing DeleteTablet for tablet cdb2a543898642d3abc3afa850178120 with delete_type TABLET_DATA_DELETED (replaced by tablet 26ac469f2b9f49efa88bbed6826ac438) from {username='slave'} at 127.0.0.1:39840
I20250625 20:01:38.104996 1400 tablet_service.cc:1515] Processing DeleteTablet for tablet cdb2a543898642d3abc3afa850178120 with delete_type TABLET_DATA_DELETED (replaced by tablet 26ac469f2b9f49efa88bbed6826ac438) from {username='slave'} at 127.0.0.1:44098
I20250625 20:01:38.105032 1401 tablet_service.cc:1515] Processing DeleteTablet for tablet 01f62ece8947463daef2b44571d4dec0 with delete_type TABLET_DATA_DELETED (replaced by tablet ad57e94ba6d048c19b763ef6cb4f0fcb) from {username='slave'} at 127.0.0.1:44098
I20250625 20:01:38.105158 1402 tablet_service.cc:1515] Processing DeleteTablet for tablet e44908760f6245a282f72cec3b5773fe with delete_type TABLET_DATA_DELETED (replaced by tablet bb3e9ff4d9c7400f889052c738a8cfda) from {username='slave'} at 127.0.0.1:44098
I20250625 20:01:38.103449 1533 tablet_service.cc:1468] Processing CreateTablet for tablet bb3e9ff4d9c7400f889052c738a8cfda (DEFAULT_TABLE table=default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:38.105379 1533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb3e9ff4d9c7400f889052c738a8cfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:38.105520 1399 tablet_service.cc:1468] Processing CreateTablet for tablet ad57e94ba6d048c19b763ef6cb4f0fcb (DEFAULT_TABLE table=default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:38.105746 1399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad57e94ba6d048c19b763ef6cb4f0fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:38.106031 3690 tablet_replica.cc:331] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:38.106146 3690 raft_consensus.cc:2241] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:38.106251 3690 raft_consensus.cc:2270] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:38.106546 3693 tablet_bootstrap.cc:492] T 26ac469f2b9f49efa88bbed6826ac438 P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:38.106626 3690 ts_tablet_manager.cc:1905] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:38.107530 3693 tablet_bootstrap.cc:654] T 26ac469f2b9f49efa88bbed6826ac438 P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:38.107890 1397 tablet_service.cc:1468] Processing CreateTablet for tablet bb3e9ff4d9c7400f889052c738a8cfda (DEFAULT_TABLE table=default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:38.108084 1397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb3e9ff4d9c7400f889052c738a8cfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:38.108192 3694 tablet_bootstrap.cc:492] T 26ac469f2b9f49efa88bbed6826ac438 P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:38.108762 3690 ts_tablet_manager.cc:1918] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:38.108821 3690 log.cc:1199] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/01f62ece8947463daef2b44571d4dec0
I20250625 20:01:38.109055 3690 ts_tablet_manager.cc:1939] T 01f62ece8947463daef2b44571d4dec0 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:38.109078 3694 tablet_bootstrap.cc:654] T 26ac469f2b9f49efa88bbed6826ac438 P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:38.109694 3690 tablet_replica.cc:331] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:38.109810 3690 raft_consensus.cc:2241] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:38.109836 3695 tablet_bootstrap.cc:492] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:38.109983 3690 raft_consensus.cc:2270] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:38.110036 1398 tablet_service.cc:1468] Processing CreateTablet for tablet 26ac469f2b9f49efa88bbed6826ac438 (DEFAULT_TABLE table=default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:38.110215 1398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ac469f2b9f49efa88bbed6826ac438. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:38.110360 3690 ts_tablet_manager.cc:1905] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:38.110700 3695 tablet_bootstrap.cc:654] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:38.110942 3691 tablet_replica.cc:331] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:38.111157 3692 tablet_replica.cc:331] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:38.111265 3692 raft_consensus.cc:2241] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:38.111378 3692 raft_consensus.cc:2270] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:38.111656 3693 tablet_bootstrap.cc:492] T 26ac469f2b9f49efa88bbed6826ac438 P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:38.111697 3692 ts_tablet_manager.cc:1905] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:38.111778 3693 ts_tablet_manager.cc:1397] T 26ac469f2b9f49efa88bbed6826ac438 P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 20:01:38.112179 3690 ts_tablet_manager.cc:1918] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:38.112233 3690 log.cc:1199] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/e44908760f6245a282f72cec3b5773fe
I20250625 20:01:38.112223 1674 tablet_service.cc:1468] Processing CreateTablet for tablet bb3e9ff4d9c7400f889052c738a8cfda (DEFAULT_TABLE table=default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:38.112210 3693 raft_consensus.cc:357] T 26ac469f2b9f49efa88bbed6826ac438 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.112308 3693 raft_consensus.cc:383] T 26ac469f2b9f49efa88bbed6826ac438 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:38.112350 3693 raft_consensus.cc:738] T 26ac469f2b9f49efa88bbed6826ac438 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:38.112439 1674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb3e9ff4d9c7400f889052c738a8cfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:38.112450 3690 ts_tablet_manager.cc:1939] T e44908760f6245a282f72cec3b5773fe P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:38.112494 3693 consensus_queue.cc:260] T 26ac469f2b9f49efa88bbed6826ac438 P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.112749 3693 ts_tablet_manager.cc:1428] T 26ac469f2b9f49efa88bbed6826ac438 P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:38.113030 3690 tablet_replica.cc:331] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df: stopping tablet replica
I20250625 20:01:38.113124 3690 raft_consensus.cc:2241] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:38.113272 3690 raft_consensus.cc:2270] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:38.113583 3690 ts_tablet_manager.cc:1905] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:38.113579 3692 ts_tablet_manager.cc:1918] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:38.113637 3692 log.cc:1199] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/cdb2a543898642d3abc3afa850178120
I20250625 20:01:38.113858 3692 ts_tablet_manager.cc:1939] T cdb2a543898642d3abc3afa850178120 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:38.114401 3692 tablet_replica.cc:331] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:38.114503 3692 raft_consensus.cc:2241] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:38.114594 3692 raft_consensus.cc:2270] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:38.114900 3692 ts_tablet_manager.cc:1905] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:38.114873 3581 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet 01f62ece8947463daef2b44571d4dec0 (table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]) successfully deleted
I20250625 20:01:38.115036 3581 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet e44908760f6245a282f72cec3b5773fe (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]) successfully deleted
I20250625 20:01:38.115321 1673 tablet_service.cc:1468] Processing CreateTablet for tablet ad57e94ba6d048c19b763ef6cb4f0fcb (DEFAULT_TABLE table=default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:01:38.115445 3690 ts_tablet_manager.cc:1918] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:38.115478 3693 tablet_bootstrap.cc:492] T bb3e9ff4d9c7400f889052c738a8cfda P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:38.115514 3690 log.cc:1199] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-1/wal/wals/cdb2a543898642d3abc3afa850178120
I20250625 20:01:38.115533 1673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad57e94ba6d048c19b763ef6cb4f0fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:01:38.115749 3690 ts_tablet_manager.cc:1939] T cdb2a543898642d3abc3afa850178120 P c73846c0665d4d53a274d053946107df: Deleting consensus metadata
I20250625 20:01:38.115821 3691 raft_consensus.cc:2241] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:01:38.116015 3691 raft_consensus.cc:2270] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 20:01:38.116335 1610 tablet.cc:2378] T 26ac469f2b9f49efa88bbed6826ac438 P c73846c0665d4d53a274d053946107df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:01:38.116819 3693 tablet_bootstrap.cc:654] T bb3e9ff4d9c7400f889052c738a8cfda P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:38.116900 3691 ts_tablet_manager.cc:1905] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:38.117668 3692 ts_tablet_manager.cc:1918] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:38.117728 3692 log.cc:1199] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/01f62ece8947463daef2b44571d4dec0
I20250625 20:01:38.117830 3584 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet cdb2a543898642d3abc3afa850178120 (table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277]) successfully deleted
I20250625 20:01:38.117942 3692 ts_tablet_manager.cc:1939] T 01f62ece8947463daef2b44571d4dec0 P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:38.118156 3581 catalog_manager.cc:4928] TS c73846c0665d4d53a274d053946107df (127.0.204.2:43561): tablet cdb2a543898642d3abc3afa850178120 (table default.table_01c13560571f419e9b6f45edee3a2a37 [id=aee290f61c8f4d7b93d4cd7192eb9277]) successfully deleted
I20250625 20:01:38.118415 3692 tablet_replica.cc:331] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08: stopping tablet replica
I20250625 20:01:38.118516 3692 raft_consensus.cc:2241] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:38.118594 3584 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet 01f62ece8947463daef2b44571d4dec0 (table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]) successfully deleted
I20250625 20:01:38.118623 3692 raft_consensus.cc:2270] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:38.118785 3694 tablet_bootstrap.cc:492] T 26ac469f2b9f49efa88bbed6826ac438 P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:38.118870 3694 ts_tablet_manager.cc:1397] T 26ac469f2b9f49efa88bbed6826ac438 P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250625 20:01:38.118925 3692 ts_tablet_manager.cc:1905] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:38.119356 3694 raft_consensus.cc:357] T 26ac469f2b9f49efa88bbed6826ac438 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
@ 0x7f51fa40ea12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250625 20:01:38.119457 3694 raft_consensus.cc:383] T 26ac469f2b9f49efa88bbed6826ac438 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:38.119484 3694 raft_consensus.cc:738] T 26ac469f2b9f49efa88bbed6826ac438 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:38.119613 3694 consensus_queue.cc:260] T 26ac469f2b9f49efa88bbed6826ac438 P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
@ 0x562bb57f5752 std::function<>::operator()() at ??:0
I20250625 20:01:38.120450 3694 ts_tablet_manager.cc:1428] T 26ac469f2b9f49efa88bbed6826ac438 P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:38.120616 3694 tablet_bootstrap.cc:492] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:38.120648 3692 ts_tablet_manager.cc:1918] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:38.120707 3692 log.cc:1199] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-0/wal/wals/e44908760f6245a282f72cec3b5773fe
I20250625 20:01:38.120903 3692 ts_tablet_manager.cc:1939] T e44908760f6245a282f72cec3b5773fe P 27aa5b53b16747f9a72485aaf2417b08: Deleting consensus metadata
I20250625 20:01:38.121496 3691 ts_tablet_manager.cc:1918] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:38.121513 3584 catalog_manager.cc:4928] TS 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): tablet e44908760f6245a282f72cec3b5773fe (table default.table_1d0eb88cbdfd4b16bb66bfb3d502672b [id=c5f1b7b4e1e8405794eea8fc65b77fb0]) successfully deleted
I20250625 20:01:38.121568 3691 log.cc:1199] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/01f62ece8947463daef2b44571d4dec0
I20250625 20:01:38.121588 3694 tablet_bootstrap.cc:654] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:38.121798 3691 ts_tablet_manager.cc:1939] T 01f62ece8947463daef2b44571d4dec0 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:38.122316 3691 tablet_replica.cc:331] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:38.122419 3691 raft_consensus.cc:2241] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:38.122442 3582 catalog_manager.cc:4928] TS 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255): tablet 01f62ece8947463daef2b44571d4dec0 (table default.table_967fd289bea3427c81a71d7bce689c47 [id=f57ce585c27c4ac3b416bc59a660690e]) successfully deleted
I20250625 20:01:38.122524 3691 raft_consensus.cc:2270] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:38.122859 3691 ts_tablet_manager.cc:1905] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f51f68c30a1 kudu::Thread::SuperviseThread() at ??:0
I20250625 20:01:38.123414 3695 tablet_bootstrap.cc:492] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:38.123493 3695 ts_tablet_manager.cc:1397] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20250625 20:01:38.123966 3695 raft_consensus.cc:357] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.124060 3695 raft_consensus.cc:383] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:38.124086 3695 raft_consensus.cc:738] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:38.124257 3695 consensus_queue.cc:260] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.124747 3691 ts_tablet_manager.cc:1918] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:38.124800 3691 log.cc:1199] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/e44908760f6245a282f72cec3b5773fe
I20250625 20:01:38.124996 3695 ts_tablet_manager.cc:1428] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:38.125008 3691 ts_tablet_manager.cc:1939] T e44908760f6245a282f72cec3b5773fe P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
I20250625 20:01:38.125787 3695 tablet_bootstrap.cc:492] T bb3e9ff4d9c7400f889052c738a8cfda P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
@ 0x7f51f70b26db start_thread at ??:0
I20250625 20:01:38.126891 3695 tablet_bootstrap.cc:654] T bb3e9ff4d9c7400f889052c738a8cfda P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:38.127435 3694 tablet_bootstrap.cc:492] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:38.127506 3694 ts_tablet_manager.cc:1397] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
@ 0x7f51f462871f clone at ??:0
I20250625 20:01:38.128298 3694 raft_consensus.cc:357] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.128373 3693 tablet_bootstrap.cc:492] T bb3e9ff4d9c7400f889052c738a8cfda P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:38.128399 3694 raft_consensus.cc:383] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:38.128433 3694 raft_consensus.cc:738] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:38.128453 3693 ts_tablet_manager.cc:1397] T bb3e9ff4d9c7400f889052c738a8cfda P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250625 20:01:38.128554 3694 consensus_queue.cc:260] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.128824 3694 ts_tablet_manager.cc:1428] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:38.128834 3691 tablet_replica.cc:331] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3: stopping tablet replica
I20250625 20:01:38.128976 3694 tablet_bootstrap.cc:492] T ad57e94ba6d048c19b763ef6cb4f0fcb P 0ed894f2ff26414190d33cd134c36bc3: Bootstrap starting.
I20250625 20:01:38.129032 3691 raft_consensus.cc:2241] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 20:01:38.129120 3691 raft_consensus.cc:2270] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:01:38.129081 3693 raft_consensus.cc:357] T bb3e9ff4d9c7400f889052c738a8cfda P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.129174 3693 raft_consensus.cc:383] T bb3e9ff4d9c7400f889052c738a8cfda P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:38.129204 3693 raft_consensus.cc:738] T bb3e9ff4d9c7400f889052c738a8cfda P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:38.129355 3693 consensus_queue.cc:260] T bb3e9ff4d9c7400f889052c738a8cfda P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.129647 3693 ts_tablet_manager.cc:1428] T bb3e9ff4d9c7400f889052c738a8cfda P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:38.129802 3693 tablet_bootstrap.cc:492] T ad57e94ba6d048c19b763ef6cb4f0fcb P c73846c0665d4d53a274d053946107df: Bootstrap starting.
I20250625 20:01:38.130687 3693 tablet_bootstrap.cc:654] T ad57e94ba6d048c19b763ef6cb4f0fcb P c73846c0665d4d53a274d053946107df: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:38.131515 3695 tablet_bootstrap.cc:492] T bb3e9ff4d9c7400f889052c738a8cfda P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:38.131613 3695 ts_tablet_manager.cc:1397] T bb3e9ff4d9c7400f889052c738a8cfda P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250625 20:01:38.131636 3693 tablet_bootstrap.cc:492] T ad57e94ba6d048c19b763ef6cb4f0fcb P c73846c0665d4d53a274d053946107df: No bootstrap required, opened a new log
I20250625 20:01:38.131748 3693 ts_tablet_manager.cc:1397] T ad57e94ba6d048c19b763ef6cb4f0fcb P c73846c0665d4d53a274d053946107df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250625 20:01:38.132107 3695 raft_consensus.cc:357] T bb3e9ff4d9c7400f889052c738a8cfda P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.132215 3695 raft_consensus.cc:383] T bb3e9ff4d9c7400f889052c738a8cfda P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:38.132161 3693 raft_consensus.cc:357] T ad57e94ba6d048c19b763ef6cb4f0fcb P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.132243 3695 raft_consensus.cc:738] T bb3e9ff4d9c7400f889052c738a8cfda P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:38.132251 3693 raft_consensus.cc:383] T ad57e94ba6d048c19b763ef6cb4f0fcb P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:38.132284 3693 raft_consensus.cc:738] T ad57e94ba6d048c19b763ef6cb4f0fcb P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73846c0665d4d53a274d053946107df, State: Initialized, Role: FOLLOWER
I20250625 20:01:38.132367 3695 consensus_queue.cc:260] T bb3e9ff4d9c7400f889052c738a8cfda P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.129866 3694 tablet_bootstrap.cc:654] T ad57e94ba6d048c19b763ef6cb4f0fcb P 0ed894f2ff26414190d33cd134c36bc3: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:38.132653 3695 ts_tablet_manager.cc:1428] T bb3e9ff4d9c7400f889052c738a8cfda P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 20:01:38.132824 3695 tablet_bootstrap.cc:492] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08: Bootstrap starting.
I20250625 20:01:38.129457 3691 ts_tablet_manager.cc:1905] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:01:38.133775 3695 tablet_bootstrap.cc:654] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08: Neither blocks nor log segments found. Creating new log.
I20250625 20:01:38.132422 3693 consensus_queue.cc:260] T ad57e94ba6d048c19b763ef6cb4f0fcb P c73846c0665d4d53a274d053946107df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.134697 3693 ts_tablet_manager.cc:1428] T ad57e94ba6d048c19b763ef6cb4f0fcb P c73846c0665d4d53a274d053946107df: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250625 20:01:38.135746 3695 tablet_bootstrap.cc:492] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08: No bootstrap required, opened a new log
I20250625 20:01:38.135820 3695 ts_tablet_manager.cc:1397] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 20:01:38.136261 3695 raft_consensus.cc:357] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.136358 3695 raft_consensus.cc:383] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:38.136387 3695 raft_consensus.cc:738] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Initialized, Role: FOLLOWER
I20250625 20:01:38.136498 3695 consensus_queue.cc:260] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.136646 3694 tablet_bootstrap.cc:492] T ad57e94ba6d048c19b763ef6cb4f0fcb P 0ed894f2ff26414190d33cd134c36bc3: No bootstrap required, opened a new log
I20250625 20:01:38.136730 3694 ts_tablet_manager.cc:1397] T ad57e94ba6d048c19b763ef6cb4f0fcb P 0ed894f2ff26414190d33cd134c36bc3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20250625 20:01:38.136759 3695 ts_tablet_manager.cc:1428] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 20:01:38.137225 3694 raft_consensus.cc:357] T ad57e94ba6d048c19b763ef6cb4f0fcb P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.137326 3694 raft_consensus.cc:383] T ad57e94ba6d048c19b763ef6cb4f0fcb P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:01:38.137360 3694 raft_consensus.cc:738] T ad57e94ba6d048c19b763ef6cb4f0fcb P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Initialized, Role: FOLLOWER
I20250625 20:01:38.137473 3694 consensus_queue.cc:260] T ad57e94ba6d048c19b763ef6cb4f0fcb P 0ed894f2ff26414190d33cd134c36bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.139618 3694 ts_tablet_manager.cc:1428] T ad57e94ba6d048c19b763ef6cb4f0fcb P 0ed894f2ff26414190d33cd134c36bc3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 20:01:38.141831 3691 ts_tablet_manager.cc:1918] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 20:01:38.141880 3691 log.cc:1199] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3: Deleting WAL directory at /tmp/dist-test-tasknmPaw0/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750881649092328-816-0/minicluster-data/ts-2/wal/wals/cdb2a543898642d3abc3afa850178120
I20250625 20:01:38.142045 3691 ts_tablet_manager.cc:1939] T cdb2a543898642d3abc3afa850178120 P 0ed894f2ff26414190d33cd134c36bc3: Deleting consensus metadata
W20250625 20:01:38.148065 1473 tablet.cc:2378] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:01:38.176033 3493 raft_consensus.cc:491] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:38.176164 3493 raft_consensus.cc:513] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.176613 3493 leader_election.cc:290] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:38.177109 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad57e94ba6d048c19b763ef6cb4f0fcb" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:38.177211 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad57e94ba6d048c19b763ef6cb4f0fcb" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:38.177284 1561 raft_consensus.cc:2466] T ad57e94ba6d048c19b763ef6cb4f0fcb P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:38.177352 1701 raft_consensus.cc:2466] T ad57e94ba6d048c19b763ef6cb4f0fcb P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:38.177696 1357 leader_election.cc:304] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:38.177906 3493 raft_consensus.cc:2802] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:38.177989 3493 raft_consensus.cc:491] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:38.178040 3493 raft_consensus.cc:3058] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:38.179291 3493 raft_consensus.cc:513] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.179765 3493 leader_election.cc:290] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:38.180084 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad57e94ba6d048c19b763ef6cb4f0fcb" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:38.180101 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad57e94ba6d048c19b763ef6cb4f0fcb" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:38.180197 1561 raft_consensus.cc:3058] T ad57e94ba6d048c19b763ef6cb4f0fcb P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:38.180225 1701 raft_consensus.cc:3058] T ad57e94ba6d048c19b763ef6cb4f0fcb P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:38.180932 1561 raft_consensus.cc:2466] T ad57e94ba6d048c19b763ef6cb4f0fcb P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:38.181200 1357 leader_election.cc:304] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:38.181222 1701 raft_consensus.cc:2466] T ad57e94ba6d048c19b763ef6cb4f0fcb P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:38.181456 3493 raft_consensus.cc:2802] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:38.181692 3493 raft_consensus.cc:695] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:38.181855 3493 consensus_queue.cc:237] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
W20250625 20:01:38.298141 1637 connection.cc:537] server connection from 127.0.0.1:43112 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:38.298633 1755 connection.cc:537] client connection to 127.0.204.62:36889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:01:38.299130 1755 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.204.62:36889)
W20250625 20:01:38.299500 1755 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.204.62:36889)
W20250625 20:01:38.299595 1755 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.204.62:36889)
W20250625 20:01:38.299685 1755 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.204.62:36889)
W20250625 20:01:38.330442 1750 tablet.cc:2378] T 26ac469f2b9f49efa88bbed6826ac438 P 0ed894f2ff26414190d33cd134c36bc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:01:38.359397 3493 raft_consensus.cc:491] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:38.359520 3493 raft_consensus.cc:513] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.359942 3493 leader_election.cc:290] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:38.360316 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ac469f2b9f49efa88bbed6826ac438" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:38.360316 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ac469f2b9f49efa88bbed6826ac438" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3" is_pre_election: true
I20250625 20:01:38.360463 1701 raft_consensus.cc:2466] T 26ac469f2b9f49efa88bbed6826ac438 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:38.360463 1561 raft_consensus.cc:2466] T 26ac469f2b9f49efa88bbed6826ac438 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 0.
I20250625 20:01:38.360740 1357 leader_election.cc:304] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:38.360908 3493 raft_consensus.cc:2802] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:38.360963 3493 raft_consensus.cc:491] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:38.360998 3493 raft_consensus.cc:3058] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:38.361929 3493 raft_consensus.cc:513] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.362267 3493 leader_election.cc:290] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Requested vote from peers c73846c0665d4d53a274d053946107df (127.0.204.2:43561), 0ed894f2ff26414190d33cd134c36bc3 (127.0.204.3:43255)
I20250625 20:01:38.362620 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ac469f2b9f49efa88bbed6826ac438" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:38.362648 1701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ac469f2b9f49efa88bbed6826ac438" candidate_uuid: "27aa5b53b16747f9a72485aaf2417b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed894f2ff26414190d33cd134c36bc3"
I20250625 20:01:38.362722 1561 raft_consensus.cc:3058] T 26ac469f2b9f49efa88bbed6826ac438 P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:38.362785 1701 raft_consensus.cc:3058] T 26ac469f2b9f49efa88bbed6826ac438 P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:38.363514 1701 raft_consensus.cc:2466] T 26ac469f2b9f49efa88bbed6826ac438 P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:38.363524 1561 raft_consensus.cc:2466] T 26ac469f2b9f49efa88bbed6826ac438 P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aa5b53b16747f9a72485aaf2417b08 in term 1.
I20250625 20:01:38.363799 1358 leader_election.cc:304] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:38.363940 3493 raft_consensus.cc:2802] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:38.364023 3493 raft_consensus.cc:695] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [term 1 LEADER]: Becoming Leader. State: Replica: 27aa5b53b16747f9a72485aaf2417b08, State: Running, Role: LEADER
I20250625 20:01:38.364153 3493 consensus_queue.cc:237] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.390623 3704 raft_consensus.cc:491] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:38.390759 3704 raft_consensus.cc:513] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.391178 3704 leader_election.cc:290] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:38.391613 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb3e9ff4d9c7400f889052c738a8cfda" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df" is_pre_election: true
I20250625 20:01:38.391625 1423 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb3e9ff4d9c7400f889052c738a8cfda" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08" is_pre_election: true
I20250625 20:01:38.391773 1561 raft_consensus.cc:2466] T bb3e9ff4d9c7400f889052c738a8cfda P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:38.391784 1423 raft_consensus.cc:2466] T bb3e9ff4d9c7400f889052c738a8cfda P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 0.
I20250625 20:01:38.392067 1634 leader_election.cc:304] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, c73846c0665d4d53a274d053946107df; no voters:
I20250625 20:01:38.392244 3704 raft_consensus.cc:2802] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:01:38.392300 3704 raft_consensus.cc:491] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:01:38.392334 3704 raft_consensus.cc:3058] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:38.393175 3704 raft_consensus.cc:513] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.393515 3704 leader_election.cc:290] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
I20250625 20:01:38.393828 1423 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb3e9ff4d9c7400f889052c738a8cfda" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aa5b53b16747f9a72485aaf2417b08"
I20250625 20:01:38.393946 1423 raft_consensus.cc:3058] T bb3e9ff4d9c7400f889052c738a8cfda P 27aa5b53b16747f9a72485aaf2417b08 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:38.394040 1561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb3e9ff4d9c7400f889052c738a8cfda" candidate_uuid: "0ed894f2ff26414190d33cd134c36bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c73846c0665d4d53a274d053946107df"
I20250625 20:01:38.394143 1561 raft_consensus.cc:3058] T bb3e9ff4d9c7400f889052c738a8cfda P c73846c0665d4d53a274d053946107df [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:01:38.394914 1423 raft_consensus.cc:2466] T bb3e9ff4d9c7400f889052c738a8cfda P 27aa5b53b16747f9a72485aaf2417b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:38.394973 1561 raft_consensus.cc:2466] T bb3e9ff4d9c7400f889052c738a8cfda P c73846c0665d4d53a274d053946107df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed894f2ff26414190d33cd134c36bc3 in term 1.
I20250625 20:01:38.395188 1637 leader_election.cc:304] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3, 27aa5b53b16747f9a72485aaf2417b08; no voters:
I20250625 20:01:38.395359 3704 raft_consensus.cc:2802] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:01:38.395447 3704 raft_consensus.cc:695] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed894f2ff26414190d33cd134c36bc3, State: Running, Role: LEADER
I20250625 20:01:38.395589 3704 consensus_queue.cc:237] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.455323 3704 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:38.455459 3704 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:38.455861 3704 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:01:38.456377 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:01:38.456523 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:01:38.456575 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:01:38.456710 3704 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:01:38.590998 3493 consensus_queue.cc:1035] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:38.595358 3715 consensus_queue.cc:1035] T ad57e94ba6d048c19b763ef6cb4f0fcb P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:38.599174 3728 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:38.599275 3728 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:38.599625 3728 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:38.600901 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:38.601279 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:38.601343 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:38.601477 3728 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:01:38.671253 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:01:38.671376 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:38.784196 3704 consensus_queue.cc:1035] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:38.787492 3704 consensus_queue.cc:1035] T bb3e9ff4d9c7400f889052c738a8cfda P 0ed894f2ff26414190d33cd134c36bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:38.925390 3715 consensus_queue.cc:1035] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:38.929137 3735 consensus_queue.cc:1035] T 26ac469f2b9f49efa88bbed6826ac438 P 27aa5b53b16747f9a72485aaf2417b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:01:39.205839 816 master-stress-test.cc:577] Killing test thread 1/25
I20250625 20:01:39.244048 3737 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:39.244184 3737 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:39.244639 3737 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:39.245811 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:39.245918 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:39.245976 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:39.246090 3737 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:01:40.180536 3752 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:40.180693 3752 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:40.181141 3752 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:40.181620 3443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20250625 20:01:40.182266 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:40.182344 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:40.182425 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:40.182555 3752 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:01:41.156962 3768 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:41.157111 3768 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:41.157553 3768 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:41.158821 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:01:41.158923 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:41.158983 3441 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:41.159108 3768 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:01:43.670300 3804 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:43.670441 3804 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:43.670881 3804 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:43.671941 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:43.672073 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
W20250625 20:01:43.672087 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:43.672178 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:43.672171 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:43.672333 3804 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:01:45.268036 3827 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:45.268163 3827 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:45.268579 3827 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:45.269129 3441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250625 20:01:45.269778 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:01:45.269881 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:45.269953 3441 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:45.270088 3827 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:01:47.073577 3857 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:47.073730 3857 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:47.074278 3857 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:01:47.075083 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:01:47.075199 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:01:47.075260 1637 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:01:47.075390 3857 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:01:48.515799 3878 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:48.515964 3878 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:48.516335 3878 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:48.517416 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:01:48.517508 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:01:48.517590 3441 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:48.517716 3878 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:01:48.672968 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:01:48.673094 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:52.003587 3925 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:52.003752 3925 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:52.004153 3925 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:52.004745 3443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250625 20:01:52.005429 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:52.005548 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:52.005618 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:52.005760 3925 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:01:52.592854 3936 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:01:52.592980 3936 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:01:52.593415 3936 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:01:52.594476 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:01:52.594596 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:01:52.594668 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:01:52.594781 3936 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:01:53.310426 3857 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:53.310570 3857 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:53.311038 3857 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:01:53.311832 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:01:53.311944 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:01:53.312033 1637 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:01:53.312166 3857 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:01:53.674134 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:01:53.674270 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:58.675268 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:01:58.675387 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:01:58.879371 3960 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:01:58.879518 3960 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:01:58.879932 3960 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:01:58.880570 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:01:58.880702 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:01:58.880785 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:01:58.880921 3960 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:02:00.616070 4047 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:02:00.616225 4047 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:02:00.616672 4047 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:02:00.617259 3443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250625 20:02:00.617916 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:02:00.618037 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:02:00.618134 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:02:00.618261 4047 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:02:03.677394 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:02:03.677543 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 20:02:06.898072 1747 heartbeater.cc:646] Failed to heartbeat to 127.0.204.60:46191 (77 consecutive failures): Network error: Failed to ping master at 127.0.204.60:46191: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:02:06.908279 4118 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:02:06.908457 4118 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:02:06.909053 4118 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:02:06.909839 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:02:06.910013 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:02:06.910116 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:02:06.910260 4118 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 20:02:06.925989 1470 heartbeater.cc:646] Failed to heartbeat to 127.0.204.60:46191 (78 consecutive failures): Network error: Failed to ping master at 127.0.204.60:46191: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:02:06.984246 1608 heartbeater.cc:646] Failed to heartbeat to 127.0.204.62:36889 (63 consecutive failures): Network error: Failed to ping master at 127.0.204.62:36889: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:02:06.984246 1607 heartbeater.cc:646] Failed to heartbeat to 127.0.204.60:46191 (75 consecutive failures): Network error: Failed to ping master at 127.0.204.60:46191: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:02:08.678514 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:02:08.678632 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:02:12.001363 4204 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:02:12.001505 4204 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:02:12.001900 4204 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:02:12.002481 3443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:02:12.003129 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:02:12.003225 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:02:12.003302 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:02:12.003451 4204 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:02:12.179154 4158 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:02:12.179299 4158 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:02:12.179776 4158 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:02:12.180475 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:02:12.180588 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:02:12.180649 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:02:12.180787 4158 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:02:13.679747 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:02:13.679879 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:02:18.680864 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:02:18.680979 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:02:23.682080 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:02:23.682212 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:02:24.485337 4360 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:02:24.485473 4360 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:02:24.485934 4360 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:02:24.486583 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:02:24.486688 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:02:24.486781 1637 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:02:24.486917 4360 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:02:28.683178 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:02:28.683298 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:02:29.349810 4428 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:02:29.349959 4428 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:02:29.350457 4428 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:02:29.350986 3443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:02:29.351545 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:02:29.351692 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:02:29.351776 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:02:29.351902 4428 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:02:33.684643 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:02:33.684778 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:02:34.398217 4467 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:02:34.398386 4467 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:02:34.398792 4467 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:02:34.399433 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:02:34.399541 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:02:34.399623 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:02:34.399808 4493 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:02:38.685729 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:02:38.685873 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:02:43.686983 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:02:43.687115 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:02:48.683307 4679 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:02:48.683442 4679 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:02:48.683948 4679 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:02:48.684399 3441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:02:48.684963 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:02:48.685045 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:02:48.685122 3441 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:02:48.685238 4679 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:02:48.687755 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:02:48.687837 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:02:53.688819 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:02:53.688943 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:02:54.071117 4714 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:02:54.071249 4714 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:02:54.071797 4714 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:02:54.072417 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:02:54.072705 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:02:54.072778 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:02:54.072921 4714 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:02:58.690199 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:02:58.690321 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:03:00.637375 4714 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:03:00.637501 4714 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:03:00.637904 4714 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:03:00.638463 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:03:00.638566 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:03:00.638661 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:03:00.638806 4714 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:03:02.035621 4854 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:03:02.035799 4854 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:03:02.036263 4854 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:03:02.036806 3443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:03:02.037489 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:03:02.037581 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:03:02.037671 3441 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:03:02.037812 4854 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:03:03.691285 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:03:03.691418 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 20:03:07.014402 1747 heartbeater.cc:646] Failed to heartbeat to 127.0.204.60:46191 (197 consecutive failures): Network error: Failed to ping master at 127.0.204.60:46191: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:03:07.040238 1471 heartbeater.cc:646] Failed to heartbeat to 127.0.204.62:36889 (184 consecutive failures): Network error: Failed to ping master at 127.0.204.62:36889: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:03:07.098380 1608 heartbeater.cc:646] Failed to heartbeat to 127.0.204.62:36889 (183 consecutive failures): Network error: Failed to ping master at 127.0.204.62:36889: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:03:08.692437 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:03:08.692605 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:03:12.727653 4860 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:03:12.727825 4860 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:03:12.728397 4860 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:03:12.728963 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:03:12.729099 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:03:12.729254 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:03:12.729434 4860 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:03:13.693475 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:03:13.693594 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:03:14.760154 4860 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:03:14.760293 4860 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:03:14.760705 4860 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:03:14.761224 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:03:14.761315 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:03:14.761408 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:03:14.761556 4860 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:03:18.694478 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:03:18.694600 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:03:20.784685 5098 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:03:20.784832 5098 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:03:20.785367 5098 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:03:20.785897 3443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:03:20.786550 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:03:20.786634 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:03:20.786712 3441 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:03:20.786831 5098 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:03:23.695531 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:03:23.695673 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:03:24.388025 5150 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:03:24.388162 5150 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:03:24.388595 5150 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:03:24.389674 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:03:24.389758 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:03:24.389835 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:03:24.389959 5150 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:03:25.396462 5121 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:03:25.396632 5121 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:03:25.397078 5121 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:03:25.397859 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:03:25.397961 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:03:25.398046 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:03:25.398186 5121 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:03:28.696760 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:03:28.696880 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:03:33.697854 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:03:33.697976 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:03:33.722666 3445 maintenance_manager.cc:419] P f87f9e18e5b948a4a3fb3666f106abd0: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033351
I20250625 20:03:33.765344 3444 maintenance_manager.cc:643] P f87f9e18e5b948a4a3fb3666f106abd0: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.042s user 0.018s sys 0.000s Metrics: {"bytes_written":27322,"cfile_init":1,"dirs.queue_time_us":241,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":953,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":477,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":70,"thread_start_us":185,"threads_started":2,"wal-append.queue_time_us":164}
I20250625 20:03:33.766196 3445 maintenance_manager.cc:419] P f87f9e18e5b948a4a3fb3666f106abd0: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 4522 bytes on disk
I20250625 20:03:33.766754 3444 maintenance_manager.cc:643] P f87f9e18e5b948a4a3fb3666f106abd0: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20250625 20:03:38.698886 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:03:38.699020 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:03:41.367393 5269 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:03:41.367558 5269 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:03:41.368105 5269 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:03:41.368774 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:03:41.368902 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:03:41.368942 1637 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:03:41.369060 5269 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:03:42.862082 5396 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:03:42.862229 5396 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:03:42.862705 5396 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:03:42.863221 3443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250625 20:03:42.863834 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:03:42.863917 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:03:42.863976 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:03:42.864094 5396 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:03:43.700160 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:03:43.700287 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:03:48.701292 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:03:48.701411 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:03:51.984458 5552 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:03:51.984603 5552 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:03:51.985038 5552 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:03:51.985621 3441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:03:51.986204 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:03:51.986333 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:03:51.986408 3441 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:03:51.986538 5552 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:03:53.702430 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:03:53.702540 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:03:58.703437 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:03:58.703564 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:03:58.929291 5607 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:03:58.929422 5607 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:03:58.929913 5607 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:03:58.930567 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:03:58.930692 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:03:58.930784 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:03:58.930956 5607 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:04:03.704627 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:04:03.704758 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:04:05.320617 5784 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:04:05.320756 5784 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:04:05.321146 5784 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:04:05.321655 3441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:04:05.322237 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:04:05.322328 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:04:05.322407 3441 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:04:05.322569 5784 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250625 20:04:07.129593 1748 heartbeater.cc:646] Failed to heartbeat to 127.0.204.62:36889 (304 consecutive failures): Network error: Failed to ping master at 127.0.204.62:36889: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:04:07.154793 1470 heartbeater.cc:646] Failed to heartbeat to 127.0.204.60:46191 (318 consecutive failures): Network error: Failed to ping master at 127.0.204.60:46191: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:04:07.215088 1608 heartbeater.cc:646] Failed to heartbeat to 127.0.204.62:36889 (303 consecutive failures): Network error: Failed to ping master at 127.0.204.62:36889: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:04:08.705770 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:04:08.705899 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:04:13.707083 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:04:13.707219 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:04:15.929205 5902 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:04:15.929370 5902 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:04:15.929942 5902 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:04:15.930685 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:04:15.930804 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:04:15.930866 1637 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:04:15.931020 5954 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:04:18.708299 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:04:18.708422 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:04:19.949680 5994 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:04:19.949851 5994 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:04:19.950268 5994 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:04:19.951030 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:04:19.951156 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:04:19.951220 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:04:19.951414 5994 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:04:22.606838 6069 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:04:22.606968 6069 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:04:22.607489 6069 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:04:22.608013 3441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:04:22.608640 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:04:22.608807 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:04:22.608897 3441 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:04:22.609040 6069 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:04:23.709375 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:04:23.709508 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:04:26.389603 6029 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:04:26.389747 6029 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:04:26.390149 6029 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:04:26.390874 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:04:26.391062 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:04:26.391156 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:04:26.391291 6029 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:04:28.711863 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:04:28.711995 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:04:28.898486 6029 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:04:28.898628 6029 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:04:28.899066 6029 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:04:28.899845 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:04:28.899962 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:04:28.900053 1637 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:04:28.900345 6029 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:04:33.712917 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:04:33.713042 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:04:38.714151 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:04:38.714277 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:04:39.138793 6346 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:04:39.138932 6346 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:04:39.139464 6346 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:04:39.139891 3441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:04:39.140611 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:04:39.140704 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:04:39.140780 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:04:39.141026 6346 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:04:43.715431 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:04:43.715566 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:04:47.186285 6437 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:04:47.186461 6437 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:04:47.186934 6437 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:04:47.187742 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:04:47.187875 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:04:47.187974 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:04:47.188241 6437 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:04:48.716449 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:04:48.716569 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:04:50.668352 6536 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:04:50.668494 6536 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:04:50.668912 6536 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:04:50.669616 3443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:04:50.670459 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:04:50.670557 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:04:50.670614 3441 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:04:50.670733 6536 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:04:53.570097 6437 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:04:53.570283 6437 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:04:53.570739 6437 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:04:53.571501 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:04:53.571601 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:04:53.571694 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:04:53.571866 6437 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:04:53.717639 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:04:53.717765 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:04:56.838640 6644 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:04:56.838783 6644 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:04:56.839272 6644 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:04:56.839933 3443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:04:56.840739 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:04:56.840844 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:04:56.840917 3441 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:04:56.841042 6644 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:04:58.718855 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:04:58.718981 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:05:02.916128 6720 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:05:02.916290 6720 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:05:02.916851 6720 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:05:02.917732 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:05:02.917857 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:05:02.917924 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:05:02.918069 6720 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:05:03.497449 6756 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:05:03.497593 6756 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:05:03.498059 6756 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:05:03.498656 3443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:05:03.499303 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:05:03.499384 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:05:03.499462 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:05:03.499610 6756 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:05:03.720041 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:05:03.720175 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:05:06.880306 6720 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:05:06.880426 6720 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:05:06.880838 6720 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:05:06.881454 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:05:06.881547 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:05:06.881630 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:05:06.881773 6720 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 20:05:07.245734 1748 heartbeater.cc:646] Failed to heartbeat to 127.0.204.62:36889 (424 consecutive failures): Network error: Failed to ping master at 127.0.204.62:36889: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:05:07.271242 1471 heartbeater.cc:646] Failed to heartbeat to 127.0.204.62:36889 (424 consecutive failures): Network error: Failed to ping master at 127.0.204.62:36889: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:05:07.335062 1608 heartbeater.cc:646] Failed to heartbeat to 127.0.204.62:36889 (423 consecutive failures): Network error: Failed to ping master at 127.0.204.62:36889: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:05:08.721239 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:05:08.721370 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:05:13.722535 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:05:13.722692 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:05:16.306627 6881 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:05:16.306772 6881 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:05:16.307132 6881 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:05:16.307806 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:05:16.307911 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:05:16.307971 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:05:16.308092 6881 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:05:18.723703 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:05:18.723897 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:05:22.985524 7071 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:05:22.985673 7071 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:05:22.986228 7071 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:05:22.986797 3443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:05:22.987442 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:05:22.987550 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:05:22.987620 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:05:22.987778 7071 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:05:23.565320 7081 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:05:23.565459 7081 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:05:23.565959 7081 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:05:23.567198 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:05:23.567308 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:05:23.567371 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:05:23.567502 7081 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:05:23.724885 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:05:23.725010 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:05:28.726169 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:05:28.726302 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:05:33.727291 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:05:33.727416 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:05:35.913486 7208 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:05:35.913630 7208 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:05:35.914160 7208 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:05:35.914978 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:05:35.915098 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:05:35.915160 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:05:35.915292 7208 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:05:37.043996 7301 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:05:37.044145 7301 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:05:37.044624 7301 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:05:37.045245 3443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250625 20:05:37.045938 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:05:37.046039 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:05:37.046113 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:05:37.046253 7301 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:05:38.728569 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:05:38.728708 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:05:43.729830 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:05:43.729960 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:05:48.730901 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:05:48.731045 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:05:53.139648 7547 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:05:53.139801 7547 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:05:53.140261 7547 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:05:53.140879 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:05:53.140995 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:05:53.141052 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:05:53.141167 7547 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:05:53.732033 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:05:53.732158 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:05:55.099607 7593 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:05:55.099778 7593 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:05:55.100297 7593 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:05:55.100833 3441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:05:55.101409 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
W20250625 20:05:55.101490 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:05:55.101572 3441 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:05:55.101706 7593 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:05:58.733225 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:05:58.733355 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:06:03.734503 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:06:03.734656 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 20:06:07.363859 1748 heartbeater.cc:646] Failed to heartbeat to 127.0.204.62:36889 (544 consecutive failures): Network error: Failed to ping master at 127.0.204.62:36889: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:06:07.388865 1470 heartbeater.cc:646] Failed to heartbeat to 127.0.204.60:46191 (558 consecutive failures): Network error: Failed to ping master at 127.0.204.60:46191: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:06:07.443336 7605 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:06:07.443472 7605 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:06:07.443871 7605 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:06:07.444444 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:06:07.444559 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:06:07.444643 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:06:07.444773 7605 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 20:06:07.451447 1608 heartbeater.cc:646] Failed to heartbeat to 127.0.204.62:36889 (543 consecutive failures): Network error: Failed to ping master at 127.0.204.62:36889: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
I20250625 20:06:08.735611 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:06:08.735759 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:06:13.737215 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:06:13.737340 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:06:14.066267 7899 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:06:14.066407 7899 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:06:14.066926 7899 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:06:14.067363 3441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:06:14.068055 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:06:14.068140 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:06:14.068197 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:06:14.068326 7899 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:06:18.738405 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:06:18.738533 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:06:20.164563 7605 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:06:20.164716 7605 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:06:20.165213 7605 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:06:20.165882 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:06:20.166009 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:06:20.166101 1634 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:06:20.166256 7605 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:06:23.739455 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:06:23.739581 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:06:28.740665 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:06:28.740803 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:06:33.625164 8208 raft_consensus.cc:491] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5852fb53503b438c92f294e0c44c916d)
I20250625 20:06:33.625279 8208 raft_consensus.cc:513] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852fb53503b438c92f294e0c44c916d" member_type: VOTER last_known_addr { host: "127.0.204.62" port: 36889 } } peers { permanent_uuid: "f87f9e18e5b948a4a3fb3666f106abd0" member_type: VOTER last_known_addr { host: "127.0.204.61" port: 38005 } } peers { permanent_uuid: "1fcdee0c0aac4013a5e3c77304b474d9" member_type: VOTER last_known_addr { host: "127.0.204.60" port: 46191 } }
I20250625 20:06:33.625789 8208 leader_election.cc:290] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889), 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191)
W20250625 20:06:33.626274 3443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 20:06:33.626829 3441 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5852fb53503b438c92f294e0c44c916d (127.0.204.62:36889): Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:06:33.626914 3443 leader_election.cc:336] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1fcdee0c0aac4013a5e3c77304b474d9 (127.0.204.60:46191): Network error: Client connection negotiation failed: client connection to 127.0.204.60:46191: connect: Connection refused (error 111)
I20250625 20:06:33.626988 3443 leader_election.cc:304] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87f9e18e5b948a4a3fb3666f106abd0; no voters: 1fcdee0c0aac4013a5e3c77304b474d9, 5852fb53503b438c92f294e0c44c916d
I20250625 20:06:33.627126 8208 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f87f9e18e5b948a4a3fb3666f106abd0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 20:06:33.741683 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:06:33.741808 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 20:06:38.262938 3445 maintenance_manager.cc:419] P f87f9e18e5b948a4a3fb3666f106abd0: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 4522 bytes on disk
I20250625 20:06:38.265461 3444 maintenance_manager.cc:643] P f87f9e18e5b948a4a3fb3666f106abd0: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.002s sys 0.000s Metrics: {"dirs.queue_time_us":164,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":167}
I20250625 20:06:38.348712 8207 raft_consensus.cc:491] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:06:38.348856 8207 raft_consensus.cc:513] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aa5b53b16747f9a72485aaf2417b08" member_type: VOTER last_known_addr { host: "127.0.204.1" port: 35759 } } peers { permanent_uuid: "c73846c0665d4d53a274d053946107df" member_type: VOTER last_known_addr { host: "127.0.204.2" port: 43561 } } peers { permanent_uuid: "0ed894f2ff26414190d33cd134c36bc3" member_type: VOTER last_known_addr { host: "127.0.204.3" port: 43255 } }
I20250625 20:06:38.349385 8207 leader_election.cc:290] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759), c73846c0665d4d53a274d053946107df (127.0.204.2:43561)
W20250625 20:06:38.350119 1634 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c73846c0665d4d53a274d053946107df (127.0.204.2:43561): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
W20250625 20:06:38.350250 1637 leader_election.cc:343] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27aa5b53b16747f9a72485aaf2417b08 (127.0.204.1:35759): Not found: Tablet not found: c00e8f8653bd42fda7bef627e99930cc
I20250625 20:06:38.350335 1637 leader_election.cc:304] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed894f2ff26414190d33cd134c36bc3; no voters: 27aa5b53b16747f9a72485aaf2417b08, c73846c0665d4d53a274d053946107df
I20250625 20:06:38.350487 8207 raft_consensus.cc:2747] T c00e8f8653bd42fda7bef627e99930cc P 0ed894f2ff26414190d33cd134c36bc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:06:38.742904 3425 master_runner.cc:186] Error getting master registration for 127.0.204.62:36889: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111),
I20250625 20:06:38.743058 3425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 20:06:38.814466 1754 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:06:38.814654 1754 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:06:38.814687 1754 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
W20250625 20:06:38.814707 1754 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
F20250625 20:06:38.825137 1789 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Aborted at 1750881998 (unix time) try "date -d @1750881998" if you are using GNU date ***
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826596 1781 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826596 1781 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.827684 1777 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800000330) received by PID 816 (TID 0x7fb61ba15700) from PID 816; stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826596 1781 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.827684 1777 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829864 1765 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_117db1e3446d41099601e6b995a36d74 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826596 1781 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.827684 1777 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829864 1765 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_117db1e3446d41099601e6b995a36d74 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829907 1767 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3ef4196a2a204be1bb572abdbef3264c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7fb63aef2980 (unknown) at ??:0
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826596 1781 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.827684 1777 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829864 1765 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_117db1e3446d41099601e6b995a36d74 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829907 1767 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3ef4196a2a204be1bb572abdbef3264c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.831027 1790 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826596 1781 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.827684 1777 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829864 1765 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_117db1e3446d41099601e6b995a36d74 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829907 1767 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3ef4196a2a204be1bb572abdbef3264c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.831027 1790 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832178 1785 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826596 1781 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.827684 1777 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829864 1765 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_117db1e3446d41099601e6b995a36d74 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829907 1767 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3ef4196a2a204be1bb572abdbef3264c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.831027 1790 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832178 1785 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832269 1782 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7fb638b04fb7 gsignal at ??:0
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826596 1781 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.827684 1777 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829864 1765 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_117db1e3446d41099601e6b995a36d74 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829907 1767 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3ef4196a2a204be1bb572abdbef3264c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.831027 1790 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832178 1785 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832269 1782 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833506 1778 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826596 1781 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.827684 1777 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829864 1765 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_117db1e3446d41099601e6b995a36d74 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829907 1767 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3ef4196a2a204be1bb572abdbef3264c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.831027 1790 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832178 1785 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832269 1782 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833506 1778 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833534 1788 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826596 1781 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.827684 1777 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829864 1765 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_117db1e3446d41099601e6b995a36d74 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829907 1767 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3ef4196a2a204be1bb572abdbef3264c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.831027 1790 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832178 1785 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832269 1782 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833506 1778 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833534 1788 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833626 1773 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bb2207463de346c3b1a4271f21eed795 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826596 1781 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.827684 1777 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829864 1765 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_117db1e3446d41099601e6b995a36d74 on the master: CreateTable timed out after deadline ex @ 0x7fb638b06921 abort
pired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829907 1767 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3ef4196a2a204be1bb572abdbef3264c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.831027 1790 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832178 1785 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832269 1782 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833506 1778 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833534 1788 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833626 1773 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bb2207463de346c3b1a4271f21eed795 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833700 1780 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826596 1781 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.827684 1777 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829864 1765 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_117db1e3446d41099601e6b995a36d74 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829907 1767 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3ef4196a2a204be1bb572abdbef3264c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.831027 1790 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832178 1785 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832269 1782 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833506 1778 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833534 1788 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833626 1773 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bb2207463de346c3b1a4271f21eed795 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833700 1780 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833762 1769 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3629c083428e4ae0a977ffb16af83bc6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.834131 1786 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.834203 1787 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation fail @ 0x7fb63996ddcd google::LogMessage::Fail()
ed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826596 1781 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.827684 1777 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829864 1765 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_117db1e3446d41099601e6b995a36d74 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829907 1767 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3ef4196a2a204be1bb572abdbef3264c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.831027 1790 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832178 1785 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832269 1782 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833506 1778 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833534 1788 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833626 1773 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bb2207463de346c3b1a4271f21eed795 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833700 1780 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833762 1769 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3629c083428e4ae0a977ffb16af83bc6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.834131 1786 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.834203 1787 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826596 1781 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.827684 1777 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829864 1765 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_117db1e3446d41099601e6b995a36d74 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829907 1767 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3ef4196a2a204be1bb572abdbef3264c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.831027 1790 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832178 1785 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832269 1782 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833506 1778 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833534 1788 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833626 1773 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bb2207463de346c3b1a4271f21eed795 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833700 1780 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833762 1769 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3629c083428e4ae0a977ffb16af83bc6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.834131 1786 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.834203 1787 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.835023 1770 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ea337ae9a0e74819be0ba8940e1828e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 20:06:38.825204 1771 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac867d9f2ac54f09a848a71cd7a01d49 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825291 1774 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d4ed5ace6c144df6af9ff3155b29c5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825336 1766 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e91ea2924c1b40fc9d7610784c293467 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825354 1783 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825300 1779 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.825206 1772 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c95f91afca114793990044d26cd96c64 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826460 1775 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a31a093a024346aab31bc8d55a00253d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826491 1784 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826515 1776 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.826596 1781 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.827684 1777 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829864 1765 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_117db1e3446d41099601e6b995a36d74 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.829907 1767 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3ef4196a2a204be1bb572abdbef3264c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.831027 1790 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832178 1785 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.832269 1782 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833506 1778 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833534 1788 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833626 1773 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bb2207463de346c3b1a4271f21eed795 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833700 1780 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.833762 1769 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3629c083428e4ae0a977ffb16af83bc6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.834131 1786 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.834203 1787 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)F20250625 20:06:38.835023 1770 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ea337ae9a0e74819be0ba8940e1828e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.204.62:36889,127.0.204.61:38005,127.0.204.60:46191, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.204.62:36889: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7fb639971b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb63996d7cc google::LogMessage::Flush() at ??:0
@ 0x7fb63996ef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x556d3d932b3d kudu::MasterStressTest::ReplaceTabletThread() at ??:0
@ 0x556d3d9447b0 std::__invoke_impl<>() at ??:0
@ 0x556d3d9424ec std::__invoke<>() at ??:0
@ 0x556d3d949b3d _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
@ 0x556d3d948b41 std::thread::_Invoker<>::operator()() at ??:0
@ 0x556d3d9478be std::thread::_State_impl<>::_M_run() at ??:0
@ 0x7fb63918c6df (unknown) at ??:0
@ 0x7fb63aee76db start_thread at ??:0
@ 0x7fb638be771f clone at ??:0