Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250628 02:04:20.820000 31512 test_util.cc:276] Using random seed: 914371604
2025-06-28T02:04:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T02:04:20Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskXgtEGt/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskXgtEGt/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/security.properties
2025-06-28 02:04:26: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskXgtEGt/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskXgtEGt/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250628 02:04:33.011739 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.62:42671
--webserver_interface=127.30.198.62
--webserver_port=0
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250628 02:04:33.099756 31753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:04:33.099921 31753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:04:33.099980 31753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:04:33.102519 31753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 02:04:33.102567 31753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:04:33.102596 31753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 02:04:33.102629 31753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 02:04:33.105839 31753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.62:42671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.198.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.31753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:04:33.106622 31753 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:04:33.107445 31753 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 02:04:33.112919 31759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:33.112941 31758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:33.112941 31761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:33.113054 31753 server_base.cc:1048] running on GCE node
I20250628 02:04:33.113444 31753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:04:33.114106 31753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:04:33.115259 31753 hybrid_clock.cc:648] HybridClock initialized: now 1751076273115241 us; error 34 us; skew 500 ppm
I20250628 02:04:33.116813 31753 webserver.cc:469] Webserver started at http://127.30.198.62:40695/ using document root <none> and password file <none>
I20250628 02:04:33.117269 31753 fs_manager.cc:362] Metadata directory not provided
I20250628 02:04:33.117322 31753 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:04:33.117481 31753 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 02:04:33.118695 31753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/instance:
uuid: "e9fd188e41da4ec0b493c144accb0344"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:33.119073 31753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal/instance:
uuid: "e9fd188e41da4ec0b493c144accb0344"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:33.121798 31753 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250628 02:04:33.123410 31767 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:33.124214 31753 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250628 02:04:33.124344 31753 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
uuid: "e9fd188e41da4ec0b493c144accb0344"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:33.124413 31753 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:04:33.136884 31753 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:04:33.137341 31753 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:04:33.137471 31753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:04:33.142974 31753 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.62:42671
I20250628 02:04:33.142984 31819 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.62:42671 every 8 connection(s)
I20250628 02:04:33.143770 31753 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/info.pb
I20250628 02:04:33.145938 31820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:33.146212 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 31753
I20250628 02:04:33.146335 31512 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal/instance
I20250628 02:04:33.147450 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.61:35059
--webserver_interface=127.30.198.61
--webserver_port=0
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250628 02:04:33.148928 31820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } has no permanent_uuid. Determining permanent_uuid...
I20250628 02:04:33.157449 31820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } has no permanent_uuid. Determining permanent_uuid...
W20250628 02:04:33.158013 31770 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:04:33.158660 31820 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:04:33.207876 31820 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } attempt: 1
W20250628 02:04:33.208983 31820 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:04:33.234151 31821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:04:33.234301 31821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:04:33.234335 31821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:04:33.237120 31821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 02:04:33.237164 31821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:04:33.237180 31821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 02:04:33.237214 31821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 02:04:33.240195 31821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.61:35059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.198.61
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.31821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:04:33.240959 31821 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:04:33.241804 31821 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 02:04:33.247176 31830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:33.247217 31829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:33.247155 31832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:33.247207 31821 server_base.cc:1048] running on GCE node
I20250628 02:04:33.247812 31821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:04:33.248487 31821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:04:33.249621 31821 hybrid_clock.cc:648] HybridClock initialized: now 1751076273249605 us; error 33 us; skew 500 ppm
I20250628 02:04:33.251236 31821 webserver.cc:469] Webserver started at http://127.30.198.61:39629/ using document root <none> and password file <none>
I20250628 02:04:33.251719 31821 fs_manager.cc:362] Metadata directory not provided
I20250628 02:04:33.251770 31821 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:04:33.251900 31821 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 02:04:33.253085 31821 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/instance:
uuid: "36911adab67246cfbfaf77ef5221fc92"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:33.253464 31821 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal/instance:
uuid: "36911adab67246cfbfaf77ef5221fc92"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:33.256197 31821 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250628 02:04:33.257807 31838 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:33.258606 31821 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:33.258700 31821 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
uuid: "36911adab67246cfbfaf77ef5221fc92"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:33.258773 31821 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:04:33.266660 31821 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:04:33.267110 31821 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:04:33.267270 31821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:04:33.273118 31821 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.61:35059
I20250628 02:04:33.273128 31890 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.61:35059 every 8 connection(s)
I20250628 02:04:33.273885 31821 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/info.pb
I20250628 02:04:33.276095 31891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:33.277141 31820 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } attempt: 2
I20250628 02:04:33.279302 31891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } has no permanent_uuid. Determining permanent_uuid...
I20250628 02:04:33.281738 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 31821
I20250628 02:04:33.281841 31512 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal/instance
I20250628 02:04:33.282527 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.60:40321
--webserver_interface=127.30.198.60
--webserver_port=0
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250628 02:04:33.284780 31891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } has no permanent_uuid. Determining permanent_uuid...
I20250628 02:04:33.284955 31820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } has no permanent_uuid. Determining permanent_uuid...
W20250628 02:04:33.286237 31820 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.30.198.60:40321: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:33.288015 31891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } has no permanent_uuid. Determining permanent_uuid...
W20250628 02:04:33.288609 31840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:04:33.289280 31891 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.30.198.60:40321: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:33.329401 31820 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } attempt: 1
W20250628 02:04:33.330518 31820 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.30.198.60:40321: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:33.338440 31891 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } attempt: 1
W20250628 02:04:33.339358 31891 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.30.198.60:40321: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:04:33.370410 31894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:04:33.370563 31894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:04:33.370616 31894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:04:33.373210 31894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 02:04:33.373260 31894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:04:33.373276 31894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 02:04:33.373309 31894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 02:04:33.376554 31894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.60:40321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.198.60
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.31894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:04:33.377290 31894 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:04:33.377676 31820 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } attempt: 2
I20250628 02:04:33.378206 31894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 02:04:33.378667 31820 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.30.198.60:40321: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:04:33.383348 31900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:33.383350 31901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:33.383359 31903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:33.383574 31894 server_base.cc:1048] running on GCE node
I20250628 02:04:33.383939 31894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:04:33.384645 31894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:04:33.385784 31894 hybrid_clock.cc:648] HybridClock initialized: now 1751076273385769 us; error 39 us; skew 500 ppm
I20250628 02:04:33.387209 31894 webserver.cc:469] Webserver started at http://127.30.198.60:35681/ using document root <none> and password file <none>
I20250628 02:04:33.387676 31894 fs_manager.cc:362] Metadata directory not provided
I20250628 02:04:33.387758 31894 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:04:33.387914 31894 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 02:04:33.389168 31894 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/instance:
uuid: "492448828d9a404fa245390eba0d56fb"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:33.389567 31894 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal/instance:
uuid: "492448828d9a404fa245390eba0d56fb"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:33.392355 31894 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250628 02:04:33.393947 31909 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:33.394748 31894 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250628 02:04:33.394847 31894 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
uuid: "492448828d9a404fa245390eba0d56fb"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:33.394922 31894 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:04:33.407496 31891 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } attempt: 2
W20250628 02:04:33.408466 31891 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.30.198.60:40321: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:33.416051 31894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:04:33.416497 31894 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:04:33.416603 31894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:04:33.422169 31894 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.60:40321
I20250628 02:04:33.422178 31961 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.60:40321 every 8 connection(s)
I20250628 02:04:33.422914 31894 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/info.pb
I20250628 02:04:33.424969 31962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:33.427737 31962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } has no permanent_uuid. Determining permanent_uuid...
I20250628 02:04:33.428226 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 31894
I20250628 02:04:33.428303 31512 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal/instance
I20250628 02:04:33.435725 31962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } has no permanent_uuid. Determining permanent_uuid...
I20250628 02:04:33.439405 31962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } has no permanent_uuid. Determining permanent_uuid...
I20250628 02:04:33.445708 31962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Bootstrap starting.
I20250628 02:04:33.447336 31962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:33.447916 31962 log.cc:826] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Log is configured to *not* fsync() on all Append() calls
I20250628 02:04:33.449110 31962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: No bootstrap required, opened a new log
I20250628 02:04:33.451293 31962 raft_consensus.cc:357] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:33.451427 31962 raft_consensus.cc:383] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:33.451500 31962 raft_consensus.cc:738] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448828d9a404fa245390eba0d56fb, State: Initialized, Role: FOLLOWER
I20250628 02:04:33.451817 31962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:33.452172 31970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:33.452306 31970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:33.452669 31962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: configured and running, proceeding with master startup.
I20250628 02:04:33.456091 31894 master_runner.cc:186] Error getting master registration for 127.30.198.62:42671: OK, instance_id { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" instance_seqno: 1751076273141532 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250628 02:04:33.457865 31894 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250628 02:04:33.458863 31984 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250628 02:04:33.458928 31984 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250628 02:04:33.485828 31820 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } attempt: 3
I20250628 02:04:33.492586 31820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Bootstrap starting.
I20250628 02:04:33.495137 31820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:33.496361 31820 log.cc:826] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Log is configured to *not* fsync() on all Append() calls
I20250628 02:04:33.498076 31820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: No bootstrap required, opened a new log
I20250628 02:04:33.501067 31820 raft_consensus.cc:357] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:33.501224 31820 raft_consensus.cc:383] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:33.501269 31820 raft_consensus.cc:738] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fd188e41da4ec0b493c144accb0344, State: Initialized, Role: FOLLOWER
I20250628 02:04:33.501523 31891 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } attempt: 3
I20250628 02:04:33.502564 31820 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:33.503662 31820 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 02:04:33.508078 31891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Bootstrap starting.
W20250628 02:04:33.509405 32002 catalog_manager.cc:1560] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250628 02:04:33.509457 32002 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250628 02:04:33.509469 31988 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:33.509588 31988 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:33.510210 31891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:33.508867 31753 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: OK, instance_id { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" instance_seqno: 1751076273271459 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250628 02:04:33.510773 31753 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:04:33.511145 31891 log.cc:826] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Log is configured to *not* fsync() on all Append() calls
I20250628 02:04:33.512564 31891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: No bootstrap required, opened a new log
I20250628 02:04:33.515980 31891 raft_consensus.cc:357] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:33.518069 31891 raft_consensus.cc:383] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:33.518169 31891 raft_consensus.cc:738] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36911adab67246cfbfaf77ef5221fc92, State: Initialized, Role: FOLLOWER
I20250628 02:04:33.518640 31891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:33.519622 31891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: configured and running, proceeding with master startup.
W20250628 02:04:33.526710 32020 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250628 02:04:33.526854 32020 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250628 02:04:33.528491 31821 master_runner.cc:263] No leader master found from master 36911adab67246cfbfaf77ef5221fc92
I20250628 02:04:33.528541 31821 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:04:33.543802 32006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:33.543982 32006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:33.585266 31988 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:33.585469 31988 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:33.586401 31988 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
I20250628 02:04:33.587221 31865 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36911adab67246cfbfaf77ef5221fc92" is_pre_election: true
I20250628 02:04:33.587466 31865 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fd188e41da4ec0b493c144accb0344 in term 0.
I20250628 02:04:33.588135 31936 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448828d9a404fa245390eba0d56fb" is_pre_election: true
I20250628 02:04:33.588343 31936 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fd188e41da4ec0b493c144accb0344 in term 0.
I20250628 02:04:33.588840 31769 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344; no voters:
I20250628 02:04:33.589385 31988 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:33.589464 31988 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:33.589507 31988 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:33.590771 31988 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:33.591054 31988 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 1 election: Requested vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
I20250628 02:04:33.591665 31936 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448828d9a404fa245390eba0d56fb"
I20250628 02:04:33.591774 31936 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:33.592608 31936 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fd188e41da4ec0b493c144accb0344 in term 1.
I20250628 02:04:33.592921 31769 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344; no voters:
I20250628 02:04:33.593149 31865 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36911adab67246cfbfaf77ef5221fc92"
I20250628 02:04:33.593235 31865 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:33.594098 31865 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fd188e41da4ec0b493c144accb0344 in term 1.
I20250628 02:04:33.594480 31988 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:33.594674 31988 raft_consensus.cc:695] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 1 LEADER]: Becoming Leader. State: Replica: e9fd188e41da4ec0b493c144accb0344, State: Running, Role: LEADER
I20250628 02:04:33.594951 31988 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:33.596572 31988 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fd188e41da4ec0b493c144accb0344. Latest consensus state: current_term: 1 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:33.596700 31988 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:33.607897 32025 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 02:04:33.608865 32025 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 02:04:33.613287 31936 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 1 FOLLOWER]: Refusing update from remote peer e9fd188e41da4ec0b493c144accb0344: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 02:04:33.613808 32023 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:33.617384 31970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fd188e41da4ec0b493c144accb0344. Latest consensus state: current_term: 1 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:33.617481 31970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:33.618255 31865 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 1 FOLLOWER]: Refusing update from remote peer e9fd188e41da4ec0b493c144accb0344: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 02:04:33.618707 32023 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:33.621946 32006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fd188e41da4ec0b493c144accb0344. Latest consensus state: current_term: 1 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:33.622056 32006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:33.625710 32023 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:33.625808 32023 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:33.626804 32006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:33.626886 32006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:33.627230 32026 mvcc.cc:204] Tried to move back new op lower bound from 7172408416714424320 to 7172408416648151040. Current Snapshot: MvccSnapshot[applied={T|T < 7172408416714424320}]
I20250628 02:04:33.627584 32029 mvcc.cc:204] Tried to move back new op lower bound from 7172408416714424320 to 7172408416648151040. Current Snapshot: MvccSnapshot[applied={T|T < 7172408416714424320}]
I20250628 02:04:33.629190 31988 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:33.629274 31988 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:33.630081 31970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:33.630167 31970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:33.634196 32027 mvcc.cc:204] Tried to move back new op lower bound from 7172408416714424320 to 7172408416648151040. Current Snapshot: MvccSnapshot[applied={T|T < 7172408416714424320}]
I20250628 02:04:33.634626 32025 catalog_manager.cc:1349] Generated new cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:33.634682 32025 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 02:04:33.644097 32025 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 02:04:33.645162 32025 catalog_manager.cc:1506] Loading token signing keys...
I20250628 02:04:33.661806 32025 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Generated new TSK 0
I20250628 02:04:33.662499 32025 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 02:04:33.663030 32025 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250628 02:04:33.663820 32025 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250628 02:04:33.674053 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.198.1:0
--local_ip_for_outbound_sockets=127.30.198.1
--webserver_interface=127.30.198.1
--webserver_port=0
--tserver_master_addrs=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250628 02:04:33.782050 32037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:04:33.782229 32037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:04:33.782303 32037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:04:33.785007 32037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:04:33.785144 32037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.198.1
I20250628 02:04:33.788203 32037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.198.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.198.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--never_fsync=true
--heap_profile_path=/tmp/kudu.32037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.198.1
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:04:33.789005 32037 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:04:33.789856 32037 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 02:04:33.796231 32042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:33.796263 32045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:33.796233 32043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:33.796380 32037 server_base.cc:1048] running on GCE node
I20250628 02:04:33.796712 32037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:04:33.797129 32037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:04:33.798269 32037 hybrid_clock.cc:648] HybridClock initialized: now 1751076273798245 us; error 33 us; skew 500 ppm
I20250628 02:04:33.799808 32037 webserver.cc:469] Webserver started at http://127.30.198.1:44145/ using document root <none> and password file <none>
I20250628 02:04:33.800256 32037 fs_manager.cc:362] Metadata directory not provided
I20250628 02:04:33.800307 32037 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:04:33.800452 32037 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 02:04:33.801661 32037 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/data/instance:
uuid: "cb3d5d42c47546258deed617114e71f9"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:33.802033 32037 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/instance:
uuid: "cb3d5d42c47546258deed617114e71f9"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:33.804692 32037 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20250628 02:04:33.806504 32051 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:33.807317 32037 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:33.807400 32037 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal
uuid: "cb3d5d42c47546258deed617114e71f9"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:33.807472 32037 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:04:33.816607 32037 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:04:33.817076 32037 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:04:33.817211 32037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:04:33.817615 32037 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 02:04:33.818316 32037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 02:04:33.818356 32037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:33.818390 32037 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 02:04:33.818416 32037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:33.825883 32037 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.1:37393
I20250628 02:04:33.825906 32164 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.1:37393 every 8 connection(s)
I20250628 02:04:33.826579 32037 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/data/info.pb
I20250628 02:04:33.829214 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 32037
I20250628 02:04:33.829322 31512 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/instance
I20250628 02:04:33.831636 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.198.2:0
--local_ip_for_outbound_sockets=127.30.198.2
--webserver_interface=127.30.198.2
--webserver_port=0
--tserver_master_addrs=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250628 02:04:33.835263 32166 heartbeater.cc:344] Connected to a master server at 127.30.198.62:42671
I20250628 02:04:33.835434 32165 heartbeater.cc:344] Connected to a master server at 127.30.198.60:40321
I20250628 02:04:33.835474 32166 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:33.835513 32165 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:33.836024 32166 heartbeater.cc:507] Master 127.30.198.62:42671 requested a full tablet report, sending...
I20250628 02:04:33.836185 32167 heartbeater.cc:344] Connected to a master server at 127.30.198.61:35059
I20250628 02:04:33.836249 32167 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:33.836392 32167 heartbeater.cc:507] Master 127.30.198.61:35059 requested a full tablet report, sending...
I20250628 02:04:33.836024 32165 heartbeater.cc:507] Master 127.30.198.60:40321 requested a full tablet report, sending...
I20250628 02:04:33.837215 31784 ts_manager.cc:194] Registered new tserver with Master: cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:33.837299 31926 ts_manager.cc:194] Registered new tserver with Master: cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:33.837291 31855 ts_manager.cc:194] Registered new tserver with Master: cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:33.838191 31784 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.198.1:42527
W20250628 02:04:33.921994 32172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:04:33.922176 32172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:04:33.922290 32172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:04:33.924939 32172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:04:33.925071 32172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.198.2
I20250628 02:04:33.928155 32172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.198.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.198.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--never_fsync=true
--heap_profile_path=/tmp/kudu.32172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.198.2
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:04:33.928901 32172 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:04:33.929728 32172 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 02:04:33.936335 32177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:33.936352 32180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:33.936691 32178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:33.936782 32172 server_base.cc:1048] running on GCE node
I20250628 02:04:33.937048 32172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:04:33.937422 32172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:04:33.938555 32172 hybrid_clock.cc:648] HybridClock initialized: now 1751076273938543 us; error 24 us; skew 500 ppm
I20250628 02:04:33.939991 32172 webserver.cc:469] Webserver started at http://127.30.198.2:40607/ using document root <none> and password file <none>
I20250628 02:04:33.940414 32172 fs_manager.cc:362] Metadata directory not provided
I20250628 02:04:33.940459 32172 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:04:33.940603 32172 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 02:04:33.941784 32172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/data/instance:
uuid: "b9085abee1ab4c96a3716383fe8fe592"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:33.942153 32172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/instance:
uuid: "b9085abee1ab4c96a3716383fe8fe592"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:33.945175 32172 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250628 02:04:33.947240 32186 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:33.948036 32172 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:33.948124 32172 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal
uuid: "b9085abee1ab4c96a3716383fe8fe592"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:33.948194 32172 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:04:33.968032 32172 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:04:33.968480 32172 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:04:33.968617 32172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:04:33.969017 32172 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 02:04:33.969712 32172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 02:04:33.969753 32172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:33.969782 32172 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 02:04:33.969800 32172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:33.977360 32172 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.2:46697
I20250628 02:04:33.977380 32299 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.2:46697 every 8 connection(s)
I20250628 02:04:33.978049 32172 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/data/info.pb
I20250628 02:04:33.985736 32302 heartbeater.cc:344] Connected to a master server at 127.30.198.61:35059
I20250628 02:04:33.985919 32302 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:33.985957 32301 heartbeater.cc:344] Connected to a master server at 127.30.198.62:42671
I20250628 02:04:33.986011 32301 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:33.986439 32300 heartbeater.cc:344] Connected to a master server at 127.30.198.60:40321
I20250628 02:04:33.986522 32300 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:33.986418 32302 heartbeater.cc:507] Master 127.30.198.61:35059 requested a full tablet report, sending...
I20250628 02:04:33.986416 32301 heartbeater.cc:507] Master 127.30.198.62:42671 requested a full tablet report, sending...
I20250628 02:04:33.986711 32300 heartbeater.cc:507] Master 127.30.198.60:40321 requested a full tablet report, sending...
I20250628 02:04:33.987402 31784 ts_manager.cc:194] Registered new tserver with Master: b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:33.987465 31855 ts_manager.cc:194] Registered new tserver with Master: b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:33.987497 31926 ts_manager.cc:194] Registered new tserver with Master: b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:33.988060 31784 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.198.2:48499
I20250628 02:04:33.988425 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 32172
I20250628 02:04:33.988507 31512 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/instance
I20250628 02:04:33.990506 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.198.3:0
--local_ip_for_outbound_sockets=127.30.198.3
--webserver_interface=127.30.198.3
--webserver_port=0
--tserver_master_addrs=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250628 02:04:34.078323 32307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:04:34.078483 32307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:04:34.078524 32307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:04:34.081064 32307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:04:34.081149 32307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.198.3
I20250628 02:04:34.084026 32307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.198.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.198.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--never_fsync=true
--heap_profile_path=/tmp/kudu.32307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.198.3
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:04:34.084795 32307 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:04:34.085677 32307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 02:04:34.091781 32312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:34.091794 32315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:34.091828 32313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:34.091928 32307 server_base.cc:1048] running on GCE node
I20250628 02:04:34.092365 32307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:04:34.092823 32307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:04:34.093963 32307 hybrid_clock.cc:648] HybridClock initialized: now 1751076274093945 us; error 32 us; skew 500 ppm
I20250628 02:04:34.095487 32307 webserver.cc:469] Webserver started at http://127.30.198.3:34309/ using document root <none> and password file <none>
I20250628 02:04:34.095980 32307 fs_manager.cc:362] Metadata directory not provided
I20250628 02:04:34.096040 32307 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:04:34.096174 32307 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 02:04:34.099171 32307 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/data/instance:
uuid: "d0aa6633f9b6481bb29b00d602764685"
format_stamp: "Formatted at 2025-06-28 02:04:34 on dist-test-slave-0t45"
I20250628 02:04:34.099534 32307 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/instance:
uuid: "d0aa6633f9b6481bb29b00d602764685"
format_stamp: "Formatted at 2025-06-28 02:04:34 on dist-test-slave-0t45"
I20250628 02:04:34.102134 32307 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:34.103755 32321 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:34.104579 32307 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:34.104663 32307 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal
uuid: "d0aa6633f9b6481bb29b00d602764685"
format_stamp: "Formatted at 2025-06-28 02:04:34 on dist-test-slave-0t45"
I20250628 02:04:34.104728 32307 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:04:34.121605 32307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:04:34.122022 32307 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:04:34.122133 32307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:04:34.122530 32307 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 02:04:34.123219 32307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 02:04:34.123266 32307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:34.123296 32307 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 02:04:34.123314 32307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:34.131183 32307 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.3:34733
I20250628 02:04:34.131214 32434 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.3:34733 every 8 connection(s)
I20250628 02:04:34.131896 32307 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/data/info.pb
I20250628 02:04:34.134960 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 32307
I20250628 02:04:34.135056 31512 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/instance
I20250628 02:04:34.140496 32436 heartbeater.cc:344] Connected to a master server at 127.30.198.62:42671
I20250628 02:04:34.140678 32436 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:34.140699 32435 heartbeater.cc:344] Connected to a master server at 127.30.198.60:40321
I20250628 02:04:34.140718 32437 heartbeater.cc:344] Connected to a master server at 127.30.198.61:35059
I20250628 02:04:34.140781 32437 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:34.140781 32435 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:34.141068 32436 heartbeater.cc:507] Master 127.30.198.62:42671 requested a full tablet report, sending...
I20250628 02:04:34.141064 32435 heartbeater.cc:507] Master 127.30.198.60:40321 requested a full tablet report, sending...
I20250628 02:04:34.141064 32437 heartbeater.cc:507] Master 127.30.198.61:35059 requested a full tablet report, sending...
I20250628 02:04:34.141885 31926 ts_manager.cc:194] Registered new tserver with Master: d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:34.141885 31855 ts_manager.cc:194] Registered new tserver with Master: d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:34.142274 31784 ts_manager.cc:194] Registered new tserver with Master: d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:34.142796 31784 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.198.3:47865
I20250628 02:04:34.154875 31512 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 02:04:34.159947 32449 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:34.160118 32449 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:34.161517 31794 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36911adab67246cfbfaf77ef5221fc92" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e9fd188e41da4ec0b493c144accb0344" is_pre_election: true
I20250628 02:04:34.161813 32450 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:34.161958 32450 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:34.162500 31936 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36911adab67246cfbfaf77ef5221fc92" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "492448828d9a404fa245390eba0d56fb" is_pre_election: true
I20250628 02:04:34.162606 31936 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36911adab67246cfbfaf77ef5221fc92 in term 1.
I20250628 02:04:34.162987 31840 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb; no voters: e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:34.163020 31865 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492448828d9a404fa245390eba0d56fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "36911adab67246cfbfaf77ef5221fc92" is_pre_election: true
I20250628 02:04:34.163101 31865 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448828d9a404fa245390eba0d56fb in term 1.
I20250628 02:04:34.163396 31912 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb; no voters:
I20250628 02:04:34.163441 32449 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
I20250628 02:04:34.163491 31794 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492448828d9a404fa245390eba0d56fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e9fd188e41da4ec0b493c144accb0344" is_pre_election: true
I20250628 02:04:34.163664 32449 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250628 02:04:34.163736 32450 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
I20250628 02:04:34.163765 32449 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:34.163846 32449 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 1 FOLLOWER]: Advancing to term 2
I20250628 02:04:34.163854 32450 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250628 02:04:34.163894 32450 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:34.163969 32450 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 1 FOLLOWER]: Advancing to term 2
I20250628 02:04:34.164896 32450 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:34.164896 32449 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:34.165241 32449 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 2 election: Requested vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
I20250628 02:04:34.165524 31865 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492448828d9a404fa245390eba0d56fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "36911adab67246cfbfaf77ef5221fc92"
I20250628 02:04:34.165568 31794 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36911adab67246cfbfaf77ef5221fc92" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e9fd188e41da4ec0b493c144accb0344"
I20250628 02:04:34.165688 31865 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 492448828d9a404fa245390eba0d56fb in current term 2: Already voted for candidate 36911adab67246cfbfaf77ef5221fc92 in this term.
I20250628 02:04:34.165731 31936 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36911adab67246cfbfaf77ef5221fc92" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "492448828d9a404fa245390eba0d56fb"
I20250628 02:04:34.165880 31936 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 36911adab67246cfbfaf77ef5221fc92 in current term 2: Already voted for candidate 492448828d9a404fa245390eba0d56fb in this term.
I20250628 02:04:34.165969 32450 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 2 election: Requested vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
I20250628 02:04:34.166010 31794 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492448828d9a404fa245390eba0d56fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e9fd188e41da4ec0b493c144accb0344"
I20250628 02:04:34.166148 31840 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:34.166285 31910 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 492448828d9a404fa245390eba0d56fb; no voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:34.166301 32451 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250628 02:04:34.166450 32450 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250628 02:04:34.168951 31512 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu with pid 31821
I20250628 02:04:34.169759 31783 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_c0096c6a343a4c4d97fa3906b65b5f2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:34.170291 31782 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_25f7e6e53154401aa745623d0a9238e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:34.170774 31781 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_48b5e32835b14bc489b2a0c3737240a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:34.171104 31780 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_aeae2af4c3b941689431955a4b4cacbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:34.172006 31779 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_0ba2136e171748f0ac70f94ad2378a15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:34.172210 31778 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_0f9f68e3f7dd46c29954d59183c6b66f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:34.172871 31777 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_51acb52242224cffa3ce3ea99517351a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:34.173302 31776 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_6296f6b4fab44283a0eb5a6ee76c4306"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:34.173686 31784 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_ccdc9aecb2d34778937160fc3d5cb4cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:34.173842 31775 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_f0ea02c8f8f944ebb5b8006e13d1f24c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250628 02:04:34.206516 31783 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c0096c6a343a4c4d97fa3906b65b5f2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250628 02:04:34.208617 31770 consensus_peers.cc:489] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 -> Peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Couldn't send request to peer 36911adab67246cfbfaf77ef5221fc92. Status: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250628 02:04:34.209033 31782 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_25f7e6e53154401aa745623d0a9238e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250628 02:04:34.209200 31784 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ccdc9aecb2d34778937160fc3d5cb4cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250628 02:04:34.209353 31776 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6296f6b4fab44283a0eb5a6ee76c4306 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250628 02:04:34.209439 31777 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_51acb52242224cffa3ce3ea99517351a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250628 02:04:34.209044 31780 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_aeae2af4c3b941689431955a4b4cacbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250628 02:04:34.209932 31775 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f0ea02c8f8f944ebb5b8006e13d1f24c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250628 02:04:34.209204 31779 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0ba2136e171748f0ac70f94ad2378a15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250628 02:04:34.209044 31781 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_48b5e32835b14bc489b2a0c3737240a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250628 02:04:34.209204 31778 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0f9f68e3f7dd46c29954d59183c6b66f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 02:04:34.273887 31936 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 2 FOLLOWER]: Rejecting Update request from peer e9fd188e41da4ec0b493c144accb0344 for earlier term 1. Current term is 2. Ops: []
I20250628 02:04:34.274380 32485 consensus_queue.cc:1046] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250628 02:04:34.274693 32485 raft_consensus.cc:3053] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 1 LEADER]: Stepping down as leader of term 1
I20250628 02:04:34.274744 32485 raft_consensus.cc:738] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e9fd188e41da4ec0b493c144accb0344, State: Running, Role: LEADER
I20250628 02:04:34.274914 32485 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:34.275417 32485 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 1 FOLLOWER]: Advancing to term 2
W20250628 02:04:34.339152 32167 heartbeater.cc:646] Failed to heartbeat to 127.30.198.61:35059 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:04:34.339546 32166 heartbeater.cc:499] Master 127.30.198.62:42671 was elected leader, sending a full tablet report...
I20250628 02:04:34.460002 31984 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:34.460068 31984 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: loading cluster ID for follower catalog manager: success
I20250628 02:04:34.462023 31984 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: acquiring CA information for follower catalog manager: success
I20250628 02:04:34.463145 31984 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250628 02:04:34.489269 32302 heartbeater.cc:646] Failed to heartbeat to 127.30.198.61:35059 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:04:34.489570 32301 heartbeater.cc:499] Master 127.30.198.62:42671 was elected leader, sending a full tablet report...
I20250628 02:04:34.510929 32002 catalog_manager.cc:1604] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250628 02:04:34.599651 31783 catalog_manager.cc:2149] created HMS catalog entry for table [id=12d3a771c18340d9a999ce700fb7ae99]
W20250628 02:04:34.600601 31783 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER
I20250628 02:04:34.624575 31782 catalog_manager.cc:2149] created HMS catalog entry for table [id=28ce113c511e41359769dba68875894a]
W20250628 02:04:34.625097 31782 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER
W20250628 02:04:34.643550 32437 heartbeater.cc:646] Failed to heartbeat to 127.30.198.61:35059 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:04:34.644085 32436 heartbeater.cc:499] Master 127.30.198.62:42671 was elected leader, sending a full tablet report...
I20250628 02:04:34.648459 31784 catalog_manager.cc:2149] created HMS catalog entry for table [id=bcc503e11564406ea896b60d0bd3122e]
W20250628 02:04:34.648821 31784 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER
I20250628 02:04:34.667454 31777 catalog_manager.cc:2149] created HMS catalog entry for table [id=c5894ab4404a447da23e515d19f957c8]
W20250628 02:04:34.667961 31777 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER
I20250628 02:04:34.691267 31780 catalog_manager.cc:2149] created HMS catalog entry for table [id=b4a6ec2441ea4b3f9429f8fa01f0bb3f]
W20250628 02:04:34.691745 31780 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER
I20250628 02:04:34.720834 31775 catalog_manager.cc:2149] created HMS catalog entry for table [id=47adc35ce6f14669b12657e82da4cfd8]
W20250628 02:04:34.721323 31775 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER
I20250628 02:04:34.740314 31779 catalog_manager.cc:2149] created HMS catalog entry for table [id=56f325f0d2044b03bda5136a7f717212]
W20250628 02:04:34.740738 31779 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER
I20250628 02:04:34.763722 31781 catalog_manager.cc:2149] created HMS catalog entry for table [id=92619a7eecd54094bd8c3e908c2314e3]
W20250628 02:04:34.764202 31781 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER
I20250628 02:04:34.786814 31776 catalog_manager.cc:2149] created HMS catalog entry for table [id=7e80cddaf6dd4ef1ba6f0b33402f9bbc]
W20250628 02:04:34.787329 31776 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER
I20250628 02:04:34.803726 31778 catalog_manager.cc:2149] created HMS catalog entry for table [id=cbce79435b8d4b95bb9db33535727e1a]
W20250628 02:04:34.804116 31778 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER
I20250628 02:04:34.834510 32492 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:34.834695 32492 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:34.835568 32492 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
I20250628 02:04:34.835616 31794 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492448828d9a404fa245390eba0d56fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e9fd188e41da4ec0b493c144accb0344" is_pre_election: true
I20250628 02:04:34.835829 31794 raft_consensus.cc:2466] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448828d9a404fa245390eba0d56fb in term 2.
W20250628 02:04:34.835978 31912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:04:34.836262 31910 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344; no voters:
I20250628 02:04:34.836513 32492 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250628 02:04:34.836570 32492 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:34.836606 32492 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 2 FOLLOWER]: Advancing to term 3
I20250628 02:04:34.837750 32492 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:34.838433 31794 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492448828d9a404fa245390eba0d56fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e9fd188e41da4ec0b493c144accb0344"
I20250628 02:04:34.838537 31794 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 2 FOLLOWER]: Advancing to term 3
I20250628 02:04:34.839567 31794 raft_consensus.cc:2466] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448828d9a404fa245390eba0d56fb in term 3.
I20250628 02:04:34.839923 31910 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344; no voters:
I20250628 02:04:34.840444 32492 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 3 election: Requested vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
I20250628 02:04:34.840580 32492 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 3 FOLLOWER]: Leader election won for term 3
I20250628 02:04:34.840698 32492 raft_consensus.cc:695] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 3 LEADER]: Becoming Leader. State: Replica: 492448828d9a404fa245390eba0d56fb, State: Running, Role: LEADER
I20250628 02:04:34.841012 32492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
W20250628 02:04:34.841457 31912 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:04:34.842079 31912 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:04:34.842617 32496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 492448828d9a404fa245390eba0d56fb. Latest consensus state: current_term: 3 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:34.842777 32496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:34.843076 32498 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 02:04:34.843899 32498 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 02:04:34.844439 32498 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:34.844502 32498 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 02:04:34.845352 32498 catalog_manager.cc:1506] Loading token signing keys...
I20250628 02:04:34.845880 32498 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Loaded TSK: 0
I20250628 02:04:34.846735 32498 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 02:04:34.847154 32498 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250628 02:04:34.847532 32498 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250628 02:04:35.129863 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.61:35059
--webserver_interface=127.30.198.61
--webserver_port=39629
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250628 02:04:35.193336 31783 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=12d3a771c18340d9a999ce700fb7ae99]
I20250628 02:04:35.193747 31783 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51330 (request call id 2) took 1024 ms. Trace:
I20250628 02:04:35.193814 31783 rpcz_store.cc:276] 0628 02:04:34.169474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0628 02:04:34.169535 (+ 61us) service_pool.cc:225] Handling call
0628 02:04:34.206691 (+ 37156us) catalog_manager.cc:2076] Acquired catalog manager lock
0628 02:04:34.207620 (+ 929us) catalog_manager.cc:2122] Created new table and tablet info
0628 02:04:34.599639 (+392019us) catalog_manager.cc:2148] Created new table in HMS catalog
0628 02:04:34.600508 (+ 869us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.600510 (+ 2us) write_op.cc:454] Released schema lock
0628 02:04:34.600671 (+ 161us) catalog_manager.cc:2156] Rolling back HMS table creation
0628 02:04:35.193727 (+593056us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0628 02:04:34.600423 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER)
0628 02:04:34.600434 (+ 11us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.600439 (+ 5us) write_op.cc:454] Released schema lock
0628 02:04:34.600441 (+ 2us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":203785,"Hive Metastore.run_cpu_time_us":679,"Hive Metastore.run_wall_time_us":780281,"child_traces":[["op",{}]]}
W20250628 02:04:35.194774 31783 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.30.198.1:42527 (request call id 2) took 854 ms (client timeout 1000 ms). Trace:
W20250628 02:04:35.194833 31783 rpcz_store.cc:269] 0628 02:04:34.339936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0628 02:04:35.194454 (+854518us) service_pool.cc:225] Handling call
0628 02:04:35.194753 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250628 02:04:35.196468 32442 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.198.62:42671)
I20250628 02:04:35.200139 31926 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52608:
name: "default.Table_c0096c6a343a4c4d97fa3906b65b5f2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:35.214202 31983 catalog_manager.cc:2747] Deleting table default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=12d3a771c18340d9a999ce700fb7ae99] in response to Hive Metastore notification log event 11
W20250628 02:04:35.214527 31983 hms_notification_log_listener.cc:329] Failed to handle Hive Metastore notification: 11 DROP_TABLE default.table_c0096c6a343a4c4d97fa3906b65b5f2d: Not found: the table does not exist: table_id: "12d3a771c18340d9a999ce700fb7ae99" table_name: "default.table_c0096c6a343a4c4d97fa3906b65b5f2d"
I20250628 02:04:35.217404 31794 raft_consensus.cc:1273] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 3 FOLLOWER]: Refusing update from remote peer 492448828d9a404fa245390eba0d56fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250628 02:04:35.218024 32492 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250628 02:04:35.220731 31912 consensus_peers.cc:489] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb -> Peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Couldn't send request to peer 36911adab67246cfbfaf77ef5221fc92. Status: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250628 02:04:35.222137 32492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:35.222258 32492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:35.224090 32504 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 492448828d9a404fa245390eba0d56fb. Latest consensus state: current_term: 3 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:35.224354 32504 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:35.224494 32504 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:35.225011 32504 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: FOLLOWER
W20250628 02:04:35.226150 31926 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c0096c6a343a4c4d97fa3906b65b5f2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 02:04:35.251591 31782 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=28ce113c511e41359769dba68875894a]
I20250628 02:04:35.251884 31782 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51330 (request call id 3) took 1082 ms. Trace:
I20250628 02:04:35.251952 31782 rpcz_store.cc:276] 0628 02:04:34.169780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0628 02:04:34.170171 (+ 391us) service_pool.cc:225] Handling call
0628 02:04:34.209106 (+ 38935us) catalog_manager.cc:2076] Acquired catalog manager lock
0628 02:04:34.209403 (+ 297us) catalog_manager.cc:2122] Created new table and tablet info
0628 02:04:34.624567 (+415164us) catalog_manager.cc:2148] Created new table in HMS catalog
0628 02:04:34.625056 (+ 489us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.625057 (+ 1us) write_op.cc:454] Released schema lock
0628 02:04:34.625170 (+ 113us) catalog_manager.cc:2156] Rolling back HMS table creation
0628 02:04:35.251864 (+626694us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0628 02:04:34.625017 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER)
0628 02:04:34.625024 (+ 7us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.625027 (+ 3us) write_op.cc:454] Released schema lock
0628 02:04:34.625029 (+ 2us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":958118,"Hive Metastore.run_cpu_time_us":352,"Hive Metastore.run_wall_time_us":83085,"spinlock_wait_cycles":11008,"child_traces":[["op",{}]]}
I20250628 02:04:35.255332 31923 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52608:
name: "default.Table_25f7e6e53154401aa745623d0a9238e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250628 02:04:35.269835 32500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:04:35.270079 32500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:04:35.270144 32500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:04:35.273159 32500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 02:04:35.273300 32500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:04:35.273377 32500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 02:04:35.273450 32500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 02:04:35.278015 32500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.61:35059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.198.61
--webserver_port=39629
--never_fsync=true
--heap_profile_path=/tmp/kudu.32500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:04:35.279366 32500 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:04:35.280694 32500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 02:04:35.293030 32515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:35.293035 32514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:35.293365 32517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:35.293772 32500 server_base.cc:1048] running on GCE node
I20250628 02:04:35.294237 32500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:04:35.295166 32500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:04:35.296382 32500 hybrid_clock.cc:648] HybridClock initialized: now 1751076275296352 us; error 32 us; skew 500 ppm
I20250628 02:04:35.298595 32500 webserver.cc:469] Webserver started at http://127.30.198.61:39629/ using document root <none> and password file <none>
I20250628 02:04:35.299108 32500 fs_manager.cc:362] Metadata directory not provided
I20250628 02:04:35.299170 32500 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:04:35.304893 32500 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20250628 02:04:35.306536 31926 catalog_manager.cc:2149] created HMS catalog entry for table [id=f244a9cbbe994d778333a82b209ffe5e]
I20250628 02:04:35.314083 31926 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52608:
table { table_name: "default.Table_c0096c6a343a4c4d97fa3906b65b5f2d" } new_table_name: "default.Table_5f71bbbfb6c04933890499752142ceb3" modify_external_catalogs: true
I20250628 02:04:35.315508 31983 catalog_manager.cc:2747] Deleting table default.table_25f7e6e53154401aa745623d0a9238e5 [id=28ce113c511e41359769dba68875894a] in response to Hive Metastore notification log event 12
W20250628 02:04:35.315647 31983 hms_notification_log_listener.cc:329] Failed to handle Hive Metastore notification: 12 DROP_TABLE default.table_25f7e6e53154401aa745623d0a9238e5: Not found: the table does not exist: table_id: "28ce113c511e41359769dba68875894a" table_name: "default.table_25f7e6e53154401aa745623d0a9238e5"
W20250628 02:04:35.321285 31923 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_25f7e6e53154401aa745623d0a9238e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 02:04:35.327296 32369 tablet_service.cc:1468] Processing CreateTablet for tablet 4b1b0f06a097474d8e631f11f0d23d40 (DEFAULT_TABLE table=default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.328184 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b1b0f06a097474d8e631f11f0d23d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.327935 32234 tablet_service.cc:1468] Processing CreateTablet for tablet 4b1b0f06a097474d8e631f11f0d23d40 (DEFAULT_TABLE table=default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.328773 32234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b1b0f06a097474d8e631f11f0d23d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.330407 32099 tablet_service.cc:1468] Processing CreateTablet for tablet 4b1b0f06a097474d8e631f11f0d23d40 (DEFAULT_TABLE table=default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.331125 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b1b0f06a097474d8e631f11f0d23d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.333700 32523 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:35.334596 32530 tablet_bootstrap.cc:492] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:35.334964 32500 fs_manager.cc:730] Time spent opening block manager: real 0.029s user 0.001s sys 0.000s
I20250628 02:04:35.335004 32531 tablet_bootstrap.cc:492] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:35.335103 32500 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
uuid: "36911adab67246cfbfaf77ef5221fc92"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:35.335616 32500 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:04:35.337878 32530 tablet_bootstrap.cc:654] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.338486 32530 log.cc:826] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685: Log is configured to *not* fsync() on all Append() calls
I20250628 02:04:35.340060 32531 tablet_bootstrap.cc:654] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.341889 32531 log.cc:826] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592: Log is configured to *not* fsync() on all Append() calls
I20250628 02:04:35.343230 32530 tablet_bootstrap.cc:492] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:35.344257 32531 tablet_bootstrap.cc:492] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:35.344499 32530 ts_tablet_manager.cc:1397] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250628 02:04:35.344775 32531 ts_tablet_manager.cc:1397] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250628 02:04:35.344835 31784 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=bcc503e11564406ea896b60d0bd3122e]
I20250628 02:04:35.345026 31784 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51330 (request call id 1) took 1175 ms. Trace:
I20250628 02:04:35.345065 31784 rpcz_store.cc:276] 0628 02:04:34.169134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0628 02:04:34.173608 (+ 4474us) service_pool.cc:225] Handling call
0628 02:04:34.209263 (+ 35655us) catalog_manager.cc:2076] Acquired catalog manager lock
0628 02:04:34.209518 (+ 255us) catalog_manager.cc:2122] Created new table and tablet info
0628 02:04:34.648453 (+438935us) catalog_manager.cc:2148] Created new table in HMS catalog
0628 02:04:34.648792 (+ 339us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.648794 (+ 2us) write_op.cc:454] Released schema lock
0628 02:04:34.648878 (+ 84us) catalog_manager.cc:2156] Rolling back HMS table creation
0628 02:04:35.345015 (+696137us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0628 02:04:34.648765 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER)
0628 02:04:34.648770 (+ 5us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.648773 (+ 3us) write_op.cc:454] Released schema lock
0628 02:04:34.648775 (+ 2us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1017507,"Hive Metastore.run_cpu_time_us":335,"Hive Metastore.run_wall_time_us":117141,"spinlock_wait_cycles":21504,"child_traces":[["op",{}]]}
I20250628 02:04:35.347290 31925 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52608:
name: "default.Table_ccdc9aecb2d34778937160fc3d5cb4cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:35.350382 32531 raft_consensus.cc:357] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:35.350875 32532 tablet_bootstrap.cc:492] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:35.353049 32530 raft_consensus.cc:357] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:35.354895 32532 tablet_bootstrap.cc:654] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.355772 32532 log.cc:826] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9: Log is configured to *not* fsync() on all Append() calls
I20250628 02:04:35.355914 32530 raft_consensus.cc:383] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.356000 32530 raft_consensus.cc:738] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.356105 32531 raft_consensus.cc:383] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.356415 32531 raft_consensus.cc:738] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.356725 32530 consensus_queue.cc:260] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:35.357226 32531 consensus_queue.cc:260] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:35.359397 31924 master_service.cc:948] ReplaceTablet: received request to replace tablet 4b1b0f06a097474d8e631f11f0d23d40 from {username='slave'} at 127.0.0.1:52608
I20250628 02:04:35.360339 31921 master_service.cc:948] ReplaceTablet: received request to replace tablet 4b1b0f06a097474d8e631f11f0d23d40 from {username='slave'} at 127.0.0.1:52608
I20250628 02:04:35.360472 31922 master_service.cc:948] ReplaceTablet: received request to replace tablet 4b1b0f06a097474d8e631f11f0d23d40 from {username='slave'} at 127.0.0.1:52608
I20250628 02:04:35.362699 31920 master_service.cc:948] ReplaceTablet: received request to replace tablet 4b1b0f06a097474d8e631f11f0d23d40 from {username='slave'} at 127.0.0.1:52608
I20250628 02:04:35.362753 31919 master_service.cc:948] ReplaceTablet: received request to replace tablet 4b1b0f06a097474d8e631f11f0d23d40 from {username='slave'} at 127.0.0.1:52608
I20250628 02:04:35.363296 32530 ts_tablet_manager.cc:1428] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20250628 02:04:35.363771 32531 ts_tablet_manager.cc:1428] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.016s user 0.004s sys 0.000s
I20250628 02:04:35.364195 32435 heartbeater.cc:499] Master 127.30.198.60:40321 was elected leader, sending a full tablet report...
I20250628 02:04:35.365342 31924 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Sending DeleteTablet for 3 replicas of tablet 4b1b0f06a097474d8e631f11f0d23d40
I20250628 02:04:35.365666 31924 catalog_manager.cc:6652] ReplaceTablet: tablet 4b1b0f06a097474d8e631f11f0d23d40 deleted and replaced by tablet 0c564f941dbf42c9a0cada48c722f024
I20250628 02:04:35.366979 32369 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b1b0f06a097474d8e631f11f0d23d40 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c564f941dbf42c9a0cada48c722f024) from {username='slave'} at 127.0.0.1:34424
I20250628 02:04:35.367309 32300 heartbeater.cc:499] Master 127.30.198.60:40321 was elected leader, sending a full tablet report...
I20250628 02:04:35.367359 32542 tablet_replica.cc:331] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:35.367563 32542 raft_consensus.cc:2241] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:35.367781 32542 raft_consensus.cc:2270] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:35.368674 32234 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b1b0f06a097474d8e631f11f0d23d40 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c564f941dbf42c9a0cada48c722f024) from {username='slave'} at 127.0.0.1:58162
I20250628 02:04:35.368744 32542 ts_tablet_manager.cc:1905] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:35.369086 32099 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b1b0f06a097474d8e631f11f0d23d40 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c564f941dbf42c9a0cada48c722f024) from {username='slave'} at 127.0.0.1:34860
I20250628 02:04:35.370857 32542 ts_tablet_manager.cc:1918] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250628 02:04:35.370965 32542 log.cc:1199] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/4b1b0f06a097474d8e631f11f0d23d40
I20250628 02:04:35.371284 32542 ts_tablet_manager.cc:1939] T 4b1b0f06a097474d8e631f11f0d23d40 P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:04:35.372416 32532 tablet_bootstrap.cc:492] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:35.372424 31911 catalog_manager.cc:4928] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733): tablet 4b1b0f06a097474d8e631f11f0d23d40 (table default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e]) successfully deleted
I20250628 02:04:35.372673 32532 ts_tablet_manager.cc:1397] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250628 02:04:35.373401 32099 tablet_service.cc:1468] Processing CreateTablet for tablet 0c564f941dbf42c9a0cada48c722f024 (DEFAULT_TABLE table=default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.373478 32369 tablet_service.cc:1468] Processing CreateTablet for tablet 0c564f941dbf42c9a0cada48c722f024 (DEFAULT_TABLE table=default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.373653 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c564f941dbf42c9a0cada48c722f024. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.373652 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c564f941dbf42c9a0cada48c722f024. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.374845 32532 raft_consensus.cc:357] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:35.374989 32532 raft_consensus.cc:383] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.375018 32532 raft_consensus.cc:738] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.375329 32532 consensus_queue.cc:260] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:35.375583 32530 tablet_bootstrap.cc:492] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:35.376147 32532 ts_tablet_manager.cc:1428] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250628 02:04:35.376444 32234 tablet_service.cc:1468] Processing CreateTablet for tablet 0c564f941dbf42c9a0cada48c722f024 (DEFAULT_TABLE table=default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.376600 32530 tablet_bootstrap.cc:654] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.376672 32234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c564f941dbf42c9a0cada48c722f024. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.378238 32531 tablet_bootstrap.cc:492] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:35.379091 32531 tablet_bootstrap.cc:654] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.379848 32543 tablet_replica.cc:331] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:35.380173 32543 raft_consensus.cc:2241] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:35.380441 32543 raft_consensus.cc:2270] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:35.381191 32165 heartbeater.cc:499] Master 127.30.198.60:40321 was elected leader, sending a full tablet report...
W20250628 02:04:35.381197 31910 catalog_manager.cc:4908] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393): delete failed for tablet 4b1b0f06a097474d8e631f11f0d23d40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b1b0f06a097474d8e631f11f0d23d40 already in progress: creating tablet
I20250628 02:04:35.381718 32543 ts_tablet_manager.cc:1905] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:35.384119 32543 ts_tablet_manager.cc:1918] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250628 02:04:35.384239 32543 log.cc:1199] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/4b1b0f06a097474d8e631f11f0d23d40
I20250628 02:04:35.385706 32099 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b1b0f06a097474d8e631f11f0d23d40 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c564f941dbf42c9a0cada48c722f024) from {username='slave'} at 127.0.0.1:34860
I20250628 02:04:35.385960 32544 tablet_replica.cc:331] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:35.386668 32543 ts_tablet_manager.cc:1939] T 4b1b0f06a097474d8e631f11f0d23d40 P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
I20250628 02:04:35.386744 32532 tablet_bootstrap.cc:492] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:35.386952 32544 raft_consensus.cc:2241] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:35.387152 32544 raft_consensus.cc:2270] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:35.387609 32532 tablet_bootstrap.cc:654] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.387756 31923 catalog_manager.cc:2149] created HMS catalog entry for table [id=9c530e20ae3f4e529c4d6936203fe1dc]
I20250628 02:04:35.388002 32544 ts_tablet_manager.cc:1905] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:35.388605 32532 tablet_bootstrap.cc:492] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:35.388696 32532 ts_tablet_manager.cc:1397] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250628 02:04:35.389209 32532 raft_consensus.cc:357] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.389302 32532 raft_consensus.cc:383] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.389333 32532 raft_consensus.cc:738] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.389472 32532 consensus_queue.cc:260] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.389710 32532 ts_tablet_manager.cc:1428] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:35.394308 31923 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52608:
table { table_name: "default.Table_25f7e6e53154401aa745623d0a9238e5" } new_table_name: "default.Table_2ffede105ab7483388016f792547fd7f" modify_external_catalogs: true
I20250628 02:04:35.397246 32369 tablet_service.cc:1468] Processing CreateTablet for tablet 6572f5ce206f447eb5e8d9ae4bd72635 (DEFAULT_TABLE table=default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.397293 32234 tablet_service.cc:1468] Processing CreateTablet for tablet 6572f5ce206f447eb5e8d9ae4bd72635 (DEFAULT_TABLE table=default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.397467 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6572f5ce206f447eb5e8d9ae4bd72635. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.397512 32234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6572f5ce206f447eb5e8d9ae4bd72635. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.397547 32531 tablet_bootstrap.cc:492] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:35.397634 32531 ts_tablet_manager.cc:1397] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20250628 02:04:35.398044 32531 raft_consensus.cc:357] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.398527 32530 tablet_bootstrap.cc:492] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:35.398610 32531 raft_consensus.cc:383] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.398684 32531 raft_consensus.cc:738] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.398799 32544 ts_tablet_manager.cc:1918] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250628 02:04:35.398861 32530 ts_tablet_manager.cc:1397] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20250628 02:04:35.398990 32531 consensus_queue.cc:260] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.399585 32530 raft_consensus.cc:357] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.399900 32531 ts_tablet_manager.cc:1428] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250628 02:04:35.400000 32530 raft_consensus.cc:383] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.400832 32531 tablet_bootstrap.cc:492] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:35.401666 32531 tablet_bootstrap.cc:654] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.402539 32531 tablet_bootstrap.cc:492] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:35.402613 32531 ts_tablet_manager.cc:1397] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250628 02:04:35.403107 32531 raft_consensus.cc:357] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } }
I20250628 02:04:35.403188 32531 raft_consensus.cc:383] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.403219 32531 raft_consensus.cc:738] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.403314 32531 consensus_queue.cc:260] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } }
I20250628 02:04:35.400079 32530 raft_consensus.cc:738] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.403537 32531 ts_tablet_manager.cc:1428] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:35.403726 32530 consensus_queue.cc:260] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.403858 32500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:04:35.404145 32530 ts_tablet_manager.cc:1428] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250628 02:04:35.404328 32530 tablet_bootstrap.cc:492] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:35.404539 32500 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:04:35.404718 32500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:04:35.408737 32530 tablet_bootstrap.cc:654] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.399471 32099 tablet_service.cc:1468] Processing CreateTablet for tablet 6572f5ce206f447eb5e8d9ae4bd72635 (DEFAULT_TABLE table=default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.409276 32544 log.cc:1199] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/4b1b0f06a097474d8e631f11f0d23d40
I20250628 02:04:35.409425 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6572f5ce206f447eb5e8d9ae4bd72635. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.409574 32544 ts_tablet_manager.cc:1939] T 4b1b0f06a097474d8e631f11f0d23d40 P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:04:35.410645 31913 catalog_manager.cc:4928] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697): tablet 4b1b0f06a097474d8e631f11f0d23d40 (table default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e]) successfully deleted
I20250628 02:04:35.411484 32532 tablet_bootstrap.cc:492] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:35.412329 32532 tablet_bootstrap.cc:654] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.412891 31910 catalog_manager.cc:4928] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393): tablet 4b1b0f06a097474d8e631f11f0d23d40 (table default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e]) successfully deleted
I20250628 02:04:35.413545 32530 tablet_bootstrap.cc:492] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:35.413619 32530 ts_tablet_manager.cc:1397] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20250628 02:04:35.413628 32532 tablet_bootstrap.cc:492] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:35.413703 32532 ts_tablet_manager.cc:1397] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250628 02:04:35.414028 32500 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.61:35059
I20250628 02:04:35.414101 32530 raft_consensus.cc:357] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } }
I20250628 02:04:35.414139 32532 raft_consensus.cc:357] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } }
I20250628 02:04:35.414196 32530 raft_consensus.cc:383] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.414222 32532 raft_consensus.cc:383] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.414247 32532 raft_consensus.cc:738] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.414242 32530 raft_consensus.cc:738] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.414345 32532 consensus_queue.cc:260] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } }
I20250628 02:04:35.414353 32530 consensus_queue.cc:260] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } }
I20250628 02:04:35.414554 32532 ts_tablet_manager.cc:1428] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250628 02:04:35.414577 32530 ts_tablet_manager.cc:1428] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250628 02:04:35.415313 32500 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/info.pb
I20250628 02:04:35.418491 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 32500
I20250628 02:04:35.419821 32599 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.61:35059 every 8 connection(s)
I20250628 02:04:35.429292 32600 sys_catalog.cc:263] Verifying existing consensus state
I20250628 02:04:35.431063 32437 heartbeater.cc:344] Connected to a master server at 127.30.198.61:35059
I20250628 02:04:35.432123 32600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Bootstrap starting.
I20250628 02:04:35.434430 32167 heartbeater.cc:344] Connected to a master server at 127.30.198.61:35059
I20250628 02:04:35.437137 32600 log.cc:826] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Log is configured to *not* fsync() on all Append() calls
I20250628 02:04:35.438712 32545 raft_consensus.cc:491] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:35.438840 32545 raft_consensus.cc:513] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.439890 32545 leader_election.cc:290] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:35.442834 32600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 02:04:35.443153 32600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Bootstrap complete.
I20250628 02:04:35.447314 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c564f941dbf42c9a0cada48c722f024" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:35.447517 32389 raft_consensus.cc:2466] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:35.447903 32053 leader_election.cc:304] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb3d5d42c47546258deed617114e71f9, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:35.448108 32545 raft_consensus.cc:2802] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:35.448171 32545 raft_consensus.cc:491] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:35.448218 32545 raft_consensus.cc:3058] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:35.449074 32545 raft_consensus.cc:513] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.449359 32545 leader_election.cc:290] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Requested vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:35.449728 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c564f941dbf42c9a0cada48c722f024" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:35.449827 32389 raft_consensus.cc:3058] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:35.450469 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c564f941dbf42c9a0cada48c722f024" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592" is_pre_election: true
I20250628 02:04:35.450738 32254 raft_consensus.cc:2466] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:35.450770 32389 raft_consensus.cc:2466] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:35.451030 32253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c564f941dbf42c9a0cada48c722f024" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592"
I20250628 02:04:35.451052 32053 leader_election.cc:304] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb3d5d42c47546258deed617114e71f9, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:35.451119 32253 raft_consensus.cc:3058] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:35.451180 32545 raft_consensus.cc:2802] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:35.451416 32545 raft_consensus.cc:695] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Becoming Leader. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Running, Role: LEADER
I20250628 02:04:35.451615 32545 consensus_queue.cc:237] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.452275 32253 raft_consensus.cc:2466] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:35.455062 31918 catalog_manager.cc:5582] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb3d5d42c47546258deed617114e71f9 (127.30.198.1). New cstate: current_term: 1 leader_uuid: "cb3d5d42c47546258deed617114e71f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } health_report { overall_health: UNKNOWN } } }
I20250628 02:04:35.457800 32600 raft_consensus.cc:357] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:35.458151 32600 raft_consensus.cc:738] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36911adab67246cfbfaf77ef5221fc92, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.458590 32600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:35.459604 32600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 02:04:35.463482 32609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:35.463579 32609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:35.467271 32623 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:35.467322 32623 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: loading cluster ID for follower catalog manager: success
I20250628 02:04:35.469341 32623 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: acquiring CA information for follower catalog manager: success
I20250628 02:04:35.470613 32623 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250628 02:04:35.479944 32303 tablet.cc:2378] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 02:04:35.502923 31777 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=c5894ab4404a447da23e515d19f957c8]
I20250628 02:04:35.503147 31777 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51330 (request call id 8) took 1330 ms. Trace:
I20250628 02:04:35.503196 31777 rpcz_store.cc:276] 0628 02:04:34.172759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0628 02:04:34.172803 (+ 44us) service_pool.cc:225] Handling call
0628 02:04:34.209501 (+ 36698us) catalog_manager.cc:2076] Acquired catalog manager lock
0628 02:04:34.209735 (+ 234us) catalog_manager.cc:2122] Created new table and tablet info
0628 02:04:34.667443 (+457708us) catalog_manager.cc:2148] Created new table in HMS catalog
0628 02:04:34.667906 (+ 463us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.667907 (+ 1us) write_op.cc:454] Released schema lock
0628 02:04:34.668018 (+ 111us) catalog_manager.cc:2156] Rolling back HMS table creation
0628 02:04:35.503132 (+835114us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0628 02:04:34.667864 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER)
0628 02:04:34.667871 (+ 7us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.667877 (+ 6us) write_op.cc:454] Released schema lock
0628 02:04:34.667879 (+ 2us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1115351,"Hive Metastore.run_cpu_time_us":346,"Hive Metastore.run_wall_time_us":177068,"spinlock_wait_cycles":2944,"child_traces":[["op",{}]]}
I20250628 02:04:35.506742 31918 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52608:
name: "default.Table_51acb52242224cffa3ce3ea99517351a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:35.512909 31919 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52608:
name: "default.Table_d40b0217227a48fc9431fb713c4ed6be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:35.570803 31920 master_service.cc:948] ReplaceTablet: received request to replace tablet 0c564f941dbf42c9a0cada48c722f024 from {username='slave'} at 127.0.0.1:52608
I20250628 02:04:35.571638 31922 master_service.cc:948] ReplaceTablet: received request to replace tablet 6572f5ce206f447eb5e8d9ae4bd72635 from {username='slave'} at 127.0.0.1:52608
I20250628 02:04:35.572224 31921 master_service.cc:948] ReplaceTablet: received request to replace tablet 0c564f941dbf42c9a0cada48c722f024 from {username='slave'} at 127.0.0.1:52608
I20250628 02:04:35.572345 31924 master_service.cc:948] ReplaceTablet: received request to replace tablet 6572f5ce206f447eb5e8d9ae4bd72635 from {username='slave'} at 127.0.0.1:52608
I20250628 02:04:35.572513 31917 master_service.cc:948] ReplaceTablet: received request to replace tablet 6572f5ce206f447eb5e8d9ae4bd72635 from {username='slave'} at 127.0.0.1:52608
I20250628 02:04:35.577513 31922 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Sending DeleteTablet for 3 replicas of tablet 6572f5ce206f447eb5e8d9ae4bd72635
I20250628 02:04:35.577778 31922 catalog_manager.cc:6652] ReplaceTablet: tablet 6572f5ce206f447eb5e8d9ae4bd72635 deleted and replaced by tablet 73d2f50340b14bd08abd193108c40b47
I20250628 02:04:35.578948 32234 tablet_service.cc:1515] Processing DeleteTablet for tablet 6572f5ce206f447eb5e8d9ae4bd72635 with delete_type TABLET_DATA_DELETED (replaced by tablet 73d2f50340b14bd08abd193108c40b47) from {username='slave'} at 127.0.0.1:58162
I20250628 02:04:35.579115 32543 tablet_replica.cc:331] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:35.579222 32543 raft_consensus.cc:2241] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:35.579314 32543 raft_consensus.cc:2270] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:35.579828 32543 ts_tablet_manager.cc:1905] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:35.580581 32099 tablet_service.cc:1515] Processing DeleteTablet for tablet 6572f5ce206f447eb5e8d9ae4bd72635 with delete_type TABLET_DATA_DELETED (replaced by tablet 73d2f50340b14bd08abd193108c40b47) from {username='slave'} at 127.0.0.1:34860
I20250628 02:04:35.580744 32544 tablet_replica.cc:331] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:35.580832 32544 raft_consensus.cc:2241] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:35.580914 32544 raft_consensus.cc:2270] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:35.581403 32544 ts_tablet_manager.cc:1905] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:35.581815 32543 ts_tablet_manager.cc:1918] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250628 02:04:35.581892 32543 log.cc:1199] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/6572f5ce206f447eb5e8d9ae4bd72635
I20250628 02:04:35.582065 32543 ts_tablet_manager.cc:1939] T 6572f5ce206f447eb5e8d9ae4bd72635 P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
W20250628 02:04:35.582300 32168 tablet.cc:2378] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 02:04:35.583180 32544 ts_tablet_manager.cc:1918] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250628 02:04:35.583236 32544 log.cc:1199] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/6572f5ce206f447eb5e8d9ae4bd72635
I20250628 02:04:35.583400 32544 ts_tablet_manager.cc:1939] T 6572f5ce206f447eb5e8d9ae4bd72635 P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:04:35.583885 31913 catalog_manager.cc:4928] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697): tablet 6572f5ce206f447eb5e8d9ae4bd72635 (table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]) successfully deleted
I20250628 02:04:35.584362 31920 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Sending DeleteTablet for 3 replicas of tablet 0c564f941dbf42c9a0cada48c722f024
I20250628 02:04:35.584424 32369 tablet_service.cc:1515] Processing DeleteTablet for tablet 6572f5ce206f447eb5e8d9ae4bd72635 with delete_type TABLET_DATA_DELETED (replaced by tablet 73d2f50340b14bd08abd193108c40b47) from {username='slave'} at 127.0.0.1:34424
I20250628 02:04:35.584578 32542 tablet_replica.cc:331] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:35.584630 31920 catalog_manager.cc:6652] ReplaceTablet: tablet 0c564f941dbf42c9a0cada48c722f024 deleted and replaced by tablet ba0c39c26dc24ee298e9790092f461cf
I20250628 02:04:35.584677 32542 raft_consensus.cc:2241] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:35.584798 32542 raft_consensus.cc:2270] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:35.585291 32542 ts_tablet_manager.cc:1905] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:35.585508 32234 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c564f941dbf42c9a0cada48c722f024 with delete_type TABLET_DATA_DELETED (replaced by tablet ba0c39c26dc24ee298e9790092f461cf) from {username='slave'} at 127.0.0.1:58162
I20250628 02:04:35.585662 32543 tablet_replica.cc:331] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:35.585740 32543 raft_consensus.cc:2241] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:35.585830 32543 raft_consensus.cc:2270] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:35.586544 32099 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c564f941dbf42c9a0cada48c722f024 with delete_type TABLET_DATA_DELETED (replaced by tablet ba0c39c26dc24ee298e9790092f461cf) from {username='slave'} at 127.0.0.1:34860
I20250628 02:04:35.586885 32543 ts_tablet_manager.cc:1905] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:35.587226 32542 ts_tablet_manager.cc:1918] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250628 02:04:35.587280 32542 log.cc:1199] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/6572f5ce206f447eb5e8d9ae4bd72635
I20250628 02:04:35.587441 32542 ts_tablet_manager.cc:1939] T 6572f5ce206f447eb5e8d9ae4bd72635 P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:04:35.588006 32369 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c564f941dbf42c9a0cada48c722f024 with delete_type TABLET_DATA_DELETED (replaced by tablet ba0c39c26dc24ee298e9790092f461cf) from {username='slave'} at 127.0.0.1:34424
I20250628 02:04:35.588068 31911 catalog_manager.cc:4928] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733): tablet 6572f5ce206f447eb5e8d9ae4bd72635 (table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]) successfully deleted
I20250628 02:04:35.588132 32542 tablet_replica.cc:331] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:35.588222 32542 raft_consensus.cc:2241] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:35.588299 32542 raft_consensus.cc:2270] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:35.588614 32542 ts_tablet_manager.cc:1905] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:35.589363 32543 ts_tablet_manager.cc:1918] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250628 02:04:35.589430 32543 log.cc:1199] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/0c564f941dbf42c9a0cada48c722f024
I20250628 02:04:35.589640 32543 ts_tablet_manager.cc:1939] T 0c564f941dbf42c9a0cada48c722f024 P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
I20250628 02:04:35.590179 32542 ts_tablet_manager.cc:1918] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250628 02:04:35.590238 31913 catalog_manager.cc:4928] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697): tablet 0c564f941dbf42c9a0cada48c722f024 (table default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e]) successfully deleted
I20250628 02:04:35.590266 32542 log.cc:1199] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/0c564f941dbf42c9a0cada48c722f024
I20250628 02:04:35.590322 32544 tablet_replica.cc:331] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:35.590406 32544 raft_consensus.cc:2241] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Raft consensus shutting down.
I20250628 02:04:35.590451 32542 ts_tablet_manager.cc:1939] T 0c564f941dbf42c9a0cada48c722f024 P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:04:35.590543 31910 catalog_manager.cc:4928] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393): tablet 6572f5ce206f447eb5e8d9ae4bd72635 (table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]) successfully deleted
I20250628 02:04:35.590648 32544 pending_rounds.cc:70] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9: Trying to abort 1 pending ops.
I20250628 02:04:35.590780 32544 pending_rounds.cc:77] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172408424252452864 op_type: NO_OP noop_request { }
I20250628 02:04:35.590888 32544 raft_consensus.cc:2887] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250628 02:04:35.590968 32544 raft_consensus.cc:2270] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:35.591717 32544 ts_tablet_manager.cc:1905] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:35.591706 31911 catalog_manager.cc:4928] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733): tablet 0c564f941dbf42c9a0cada48c722f024 (table default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e]) successfully deleted
I20250628 02:04:35.592679 31921 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52608:
name: "default.Table_48852943c5c64ddfbde3e614d034c8d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:35.593406 32544 ts_tablet_manager.cc:1918] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:35.593482 32544 log.cc:1199] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/0c564f941dbf42c9a0cada48c722f024
I20250628 02:04:35.593703 32544 ts_tablet_manager.cc:1939] T 0c564f941dbf42c9a0cada48c722f024 P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:04:35.594883 32234 tablet_service.cc:1468] Processing CreateTablet for tablet 73d2f50340b14bd08abd193108c40b47 (DEFAULT_TABLE table=default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.595110 32234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73d2f50340b14bd08abd193108c40b47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.595248 31910 catalog_manager.cc:4928] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393): tablet 0c564f941dbf42c9a0cada48c722f024 (table default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e]) successfully deleted
I20250628 02:04:35.595304 32369 tablet_service.cc:1468] Processing CreateTablet for tablet 73d2f50340b14bd08abd193108c40b47 (DEFAULT_TABLE table=default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.595525 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73d2f50340b14bd08abd193108c40b47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.595899 32099 tablet_service.cc:1468] Processing CreateTablet for tablet 73d2f50340b14bd08abd193108c40b47 (DEFAULT_TABLE table=default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.596096 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73d2f50340b14bd08abd193108c40b47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.597337 32530 tablet_bootstrap.cc:492] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:35.597666 32532 tablet_bootstrap.cc:492] T 73d2f50340b14bd08abd193108c40b47 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:35.598155 32530 tablet_bootstrap.cc:654] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.598428 32532 tablet_bootstrap.cc:654] T 73d2f50340b14bd08abd193108c40b47 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.600822 32531 tablet_bootstrap.cc:492] T 73d2f50340b14bd08abd193108c40b47 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:35.601855 32531 tablet_bootstrap.cc:654] T 73d2f50340b14bd08abd193108c40b47 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.604995 32369 tablet_service.cc:1468] Processing CreateTablet for tablet ba0c39c26dc24ee298e9790092f461cf (DEFAULT_TABLE table=default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.605012 32234 tablet_service.cc:1468] Processing CreateTablet for tablet ba0c39c26dc24ee298e9790092f461cf (DEFAULT_TABLE table=default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.605221 32234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba0c39c26dc24ee298e9790092f461cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.605240 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba0c39c26dc24ee298e9790092f461cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.605902 32530 tablet_bootstrap.cc:492] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:35.606016 32530 ts_tablet_manager.cc:1397] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250628 02:04:35.606472 32532 tablet_bootstrap.cc:492] T 73d2f50340b14bd08abd193108c40b47 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:35.606426 32530 raft_consensus.cc:357] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.606514 32530 raft_consensus.cc:383] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.606541 32530 raft_consensus.cc:738] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.606544 32532 ts_tablet_manager.cc:1397] T 73d2f50340b14bd08abd193108c40b47 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250628 02:04:35.606639 32530 consensus_queue.cc:260] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.606841 32530 ts_tablet_manager.cc:1428] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250628 02:04:35.606895 32532 raft_consensus.cc:357] T 73d2f50340b14bd08abd193108c40b47 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.606976 32532 raft_consensus.cc:383] T 73d2f50340b14bd08abd193108c40b47 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.606999 32530 tablet_bootstrap.cc:492] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:35.607000 32532 raft_consensus.cc:738] T 73d2f50340b14bd08abd193108c40b47 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.607093 32532 consensus_queue.cc:260] T 73d2f50340b14bd08abd193108c40b47 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.607286 32532 ts_tablet_manager.cc:1428] T 73d2f50340b14bd08abd193108c40b47 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:35.607992 32530 tablet_bootstrap.cc:654] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.608814 32560 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cb3d5d42c47546258deed617114e71f9" instance_seqno: 1751076273823804) as {username='slave'} at 127.30.198.1:44213; Asking this server to re-register.
I20250628 02:04:35.608940 32558 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" instance_seqno: 1751076274129070) as {username='slave'} at 127.30.198.3:35609; Asking this server to re-register.
I20250628 02:04:35.609241 32167 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:35.609760 32437 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:35.611055 32558 ts_manager.cc:194] Registered new tserver with Master: cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:35.611310 32560 ts_manager.cc:194] Registered new tserver with Master: d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:35.612634 32531 tablet_bootstrap.cc:492] T 73d2f50340b14bd08abd193108c40b47 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:35.612712 32531 ts_tablet_manager.cc:1397] T 73d2f50340b14bd08abd193108c40b47 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20250628 02:04:35.613112 32531 raft_consensus.cc:357] T 73d2f50340b14bd08abd193108c40b47 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.613188 32531 raft_consensus.cc:383] T 73d2f50340b14bd08abd193108c40b47 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.613224 32531 raft_consensus.cc:738] T 73d2f50340b14bd08abd193108c40b47 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.613325 32531 consensus_queue.cc:260] T 73d2f50340b14bd08abd193108c40b47 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.613543 32531 ts_tablet_manager.cc:1428] T 73d2f50340b14bd08abd193108c40b47 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250628 02:04:35.613678 32531 tablet_bootstrap.cc:492] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:35.613723 32099 tablet_service.cc:1468] Processing CreateTablet for tablet ba0c39c26dc24ee298e9790092f461cf (DEFAULT_TABLE table=default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.613899 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba0c39c26dc24ee298e9790092f461cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.614410 32531 tablet_bootstrap.cc:654] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.618546 32530 tablet_bootstrap.cc:492] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:35.618623 32530 ts_tablet_manager.cc:1397] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20250628 02:04:35.618984 32530 raft_consensus.cc:357] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.619076 32530 raft_consensus.cc:383] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.619102 32530 raft_consensus.cc:738] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.619196 32530 consensus_queue.cc:260] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.619393 32530 ts_tablet_manager.cc:1428] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250628 02:04:35.619493 32532 tablet_bootstrap.cc:492] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:35.620299 32532 tablet_bootstrap.cc:654] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.621953 32532 tablet_bootstrap.cc:492] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:35.622022 32532 ts_tablet_manager.cc:1397] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250628 02:04:35.622381 32532 raft_consensus.cc:357] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.622486 32532 raft_consensus.cc:383] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.622520 32532 raft_consensus.cc:738] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.622613 32532 consensus_queue.cc:260] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.622814 32532 ts_tablet_manager.cc:1428] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:35.624079 32302 heartbeater.cc:344] Connected to a master server at 127.30.198.61:35059
I20250628 02:04:35.625553 32558 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" instance_seqno: 1751076273975274) as {username='slave'} at 127.30.198.2:33355; Asking this server to re-register.
I20250628 02:04:35.626207 32302 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:35.626854 32560 ts_manager.cc:194] Registered new tserver with Master: b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:35.628355 32531 tablet_bootstrap.cc:492] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:35.628439 32531 ts_tablet_manager.cc:1397] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20250628 02:04:35.628819 32531 raft_consensus.cc:357] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.628921 32531 raft_consensus.cc:383] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.628957 32531 raft_consensus.cc:738] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.629091 32531 consensus_queue.cc:260] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.630985 32531 ts_tablet_manager.cc:1428] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250628 02:04:35.634264 31780 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=b4a6ec2441ea4b3f9429f8fa01f0bb3f]
I20250628 02:04:35.634436 31780 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51330 (request call id 5) took 1463 ms. Trace:
I20250628 02:04:35.634491 31780 rpcz_store.cc:276] 0628 02:04:34.170993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0628 02:04:34.171030 (+ 37us) service_pool.cc:225] Handling call
0628 02:04:34.209674 (+ 38644us) catalog_manager.cc:2076] Acquired catalog manager lock
0628 02:04:34.209927 (+ 253us) catalog_manager.cc:2122] Created new table and tablet info
0628 02:04:34.691259 (+481332us) catalog_manager.cc:2148] Created new table in HMS catalog
0628 02:04:34.691667 (+ 408us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.691669 (+ 2us) write_op.cc:454] Released schema lock
0628 02:04:34.691808 (+ 139us) catalog_manager.cc:2156] Rolling back HMS table creation
0628 02:04:35.634425 (+942617us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0628 02:04:34.691631 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER)
0628 02:04:34.691637 (+ 6us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.691640 (+ 3us) write_op.cc:454] Released schema lock
0628 02:04:34.691643 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1268500,"Hive Metastore.run_cpu_time_us":392,"Hive Metastore.run_wall_time_us":155006,"spinlock_wait_cycles":4480,"child_traces":[["op",{}]]}
W20250628 02:04:35.636224 32438 tablet.cc:2378] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 02:04:35.638244 31917 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52608:
name: "default.Table_aeae2af4c3b941689431955a4b4cacbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:35.651273 31926 catalog_manager.cc:3460] renamed table default.table_c0096c6a343a4c4d97fa3906b65b5f2d [id=f244a9cbbe994d778333a82b209ffe5e] in HMS: new name default.table_5f71bbbfb6c04933890499752142ceb3
I20250628 02:04:35.657001 31983 catalog_manager.cc:2747] Deleting table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=bcc503e11564406ea896b60d0bd3122e] in response to Hive Metastore notification log event 14
W20250628 02:04:35.657168 31983 hms_notification_log_listener.cc:329] Failed to handle Hive Metastore notification: 14 DROP_TABLE default.table_ccdc9aecb2d34778937160fc3d5cb4cc: Not found: the table does not exist: table_id: "bcc503e11564406ea896b60d0bd3122e" table_name: "default.table_ccdc9aecb2d34778937160fc3d5cb4cc"
I20250628 02:04:35.657487 31983 catalog_manager.cc:2747] Deleting table default.table_51acb52242224cffa3ce3ea99517351a [id=c5894ab4404a447da23e515d19f957c8] in response to Hive Metastore notification log event 16
W20250628 02:04:35.657606 31983 hms_notification_log_listener.cc:329] Failed to handle Hive Metastore notification: 16 DROP_TABLE default.table_51acb52242224cffa3ce3ea99517351a: Not found: the table does not exist: table_id: "c5894ab4404a447da23e515d19f957c8" table_name: "default.table_51acb52242224cffa3ce3ea99517351a"
I20250628 02:04:35.657889 31983 catalog_manager.cc:2747] Deleting table default.table_aeae2af4c3b941689431955a4b4cacbd [id=b4a6ec2441ea4b3f9429f8fa01f0bb3f] in response to Hive Metastore notification log event 17
W20250628 02:04:35.657991 31983 hms_notification_log_listener.cc:329] Failed to handle Hive Metastore notification: 17 DROP_TABLE default.table_aeae2af4c3b941689431955a4b4cacbd: Not found: the table does not exist: table_id: "b4a6ec2441ea4b3f9429f8fa01f0bb3f" table_name: "default.table_aeae2af4c3b941689431955a4b4cacbd"
I20250628 02:04:35.661576 32607 raft_consensus.cc:491] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:35.661679 32607 raft_consensus.cc:513] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.662103 32607 leader_election.cc:290] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:35.662765 32253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba0c39c26dc24ee298e9790092f461cf" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592" is_pre_election: true
I20250628 02:04:35.662854 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba0c39c26dc24ee298e9790092f461cf" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:35.662899 32253 raft_consensus.cc:2466] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:35.662983 32389 raft_consensus.cc:2466] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:35.663252 32055 leader_election.cc:304] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:35.664129 32607 raft_consensus.cc:2802] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:35.664207 32607 raft_consensus.cc:491] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:35.664252 32607 raft_consensus.cc:3058] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:35.665709 32607 raft_consensus.cc:513] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.666071 32607 leader_election.cc:290] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Requested vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:35.666528 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba0c39c26dc24ee298e9790092f461cf" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:35.666638 32389 raft_consensus.cc:3058] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:35.667598 32389 raft_consensus.cc:2466] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:35.668061 32053 leader_election.cc:304] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb3d5d42c47546258deed617114e71f9, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:35.668586 32253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba0c39c26dc24ee298e9790092f461cf" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592"
I20250628 02:04:35.668709 32253 raft_consensus.cc:3058] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:35.670336 32253 raft_consensus.cc:2466] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:35.672307 32607 raft_consensus.cc:2802] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:35.672396 32607 raft_consensus.cc:695] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Becoming Leader. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Running, Role: LEADER
I20250628 02:04:35.672536 32607 consensus_queue.cc:237] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
W20250628 02:04:35.673645 31983 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba0c39c26dc24ee298e9790092f461cf consensus state has no leader
W20250628 02:04:35.677678 31925 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ccdc9aecb2d34778937160fc3d5cb4cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 02:04:35.677767 31920 catalog_manager.cc:5582] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb3d5d42c47546258deed617114e71f9 (127.30.198.1). New cstate: current_term: 1 leader_uuid: "cb3d5d42c47546258deed617114e71f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } health_report { overall_health: UNKNOWN } } }
I20250628 02:04:35.677964 31920 catalog_manager.cc:5674] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393) does not have the latest schema for tablet ba0c39c26dc24ee298e9790092f461cf (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]). Expected version 1 got 0
I20250628 02:04:35.680622 31775 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=47adc35ce6f14669b12657e82da4cfd8]
I20250628 02:04:35.680823 31775 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51330 (request call id 10) took 1507 ms. Trace:
I20250628 02:04:35.680886 31775 rpcz_store.cc:276] 0628 02:04:34.173722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0628 02:04:34.173762 (+ 40us) service_pool.cc:225] Handling call
0628 02:04:34.209987 (+ 36225us) catalog_manager.cc:2076] Acquired catalog manager lock
0628 02:04:34.210287 (+ 300us) catalog_manager.cc:2122] Created new table and tablet info
0628 02:04:34.720825 (+510538us) catalog_manager.cc:2148] Created new table in HMS catalog
0628 02:04:34.721282 (+ 457us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.721284 (+ 2us) write_op.cc:454] Released schema lock
0628 02:04:34.721400 (+ 116us) catalog_manager.cc:2156] Rolling back HMS table creation
0628 02:04:35.680811 (+959411us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0628 02:04:34.721241 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER)
0628 02:04:34.721248 (+ 7us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.721253 (+ 5us) write_op.cc:454] Released schema lock
0628 02:04:34.721255 (+ 2us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1393612,"Hive Metastore.run_cpu_time_us":315,"Hive Metastore.run_wall_time_us":75954,"child_traces":[["op",{}]]}
I20250628 02:04:35.683979 31920 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52608:
name: "default.Table_f0ea02c8f8f944ebb5b8006e13d1f24c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:35.688541 31983 catalog_manager.cc:2747] Deleting table default.table_f0ea02c8f8f944ebb5b8006e13d1f24c [id=47adc35ce6f14669b12657e82da4cfd8] in response to Hive Metastore notification log event 19
W20250628 02:04:35.688674 31983 hms_notification_log_listener.cc:329] Failed to handle Hive Metastore notification: 19 DROP_TABLE default.table_f0ea02c8f8f944ebb5b8006e13d1f24c: Not found: the table does not exist: table_id: "47adc35ce6f14669b12657e82da4cfd8" table_name: "default.table_f0ea02c8f8f944ebb5b8006e13d1f24c"
W20250628 02:04:35.693032 31918 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_51acb52242224cffa3ce3ea99517351a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250628 02:04:35.693729 31919 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d40b0217227a48fc9431fb713c4ed6be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250628 02:04:35.694366 31921 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_48852943c5c64ddfbde3e614d034c8d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250628 02:04:35.695042 31917 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_aeae2af4c3b941689431955a4b4cacbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 02:04:35.696792 31926 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52608:
table { table_name: "default.Table_5f71bbbfb6c04933890499752142ceb3" } new_table_name: "default.Table_4d41a145f10944dca41a0675451382fc" modify_external_catalogs: true
I20250628 02:04:35.727622 31925 catalog_manager.cc:2149] created HMS catalog entry for table [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]
I20250628 02:04:35.728498 32573 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 2 FOLLOWER]: Advancing to term 3
I20250628 02:04:35.732578 31925 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52608:
table { table_name: "default.Table_ccdc9aecb2d34778937160fc3d5cb4cc" } modify_external_catalogs: true
I20250628 02:04:35.732708 31925 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52608:
table { table_name: "default.Table_ccdc9aecb2d34778937160fc3d5cb4cc" } modify_external_catalogs: true
I20250628 02:04:35.736802 32609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 492448828d9a404fa245390eba0d56fb. Latest consensus state: current_term: 3 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:35.736905 32609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:35.736981 32641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:35.737047 32641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:35.737303 32369 tablet_service.cc:1468] Processing CreateTablet for tablet 48ccc4cdd28b422ca19674eb5285230d (DEFAULT_TABLE table=default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.737303 32234 tablet_service.cc:1468] Processing CreateTablet for tablet 48ccc4cdd28b422ca19674eb5285230d (DEFAULT_TABLE table=default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.737329 32099 tablet_service.cc:1468] Processing CreateTablet for tablet 48ccc4cdd28b422ca19674eb5285230d (DEFAULT_TABLE table=default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.737557 32234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48ccc4cdd28b422ca19674eb5285230d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.737557 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48ccc4cdd28b422ca19674eb5285230d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.737565 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48ccc4cdd28b422ca19674eb5285230d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.738955 32532 tablet_bootstrap.cc:492] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:35.739342 32531 tablet_bootstrap.cc:492] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:35.739342 32530 tablet_bootstrap.cc:492] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:35.739977 32532 tablet_bootstrap.cc:654] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.740212 32531 tablet_bootstrap.cc:654] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.740218 32530 tablet_bootstrap.cc:654] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.741629 32492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:35.741737 32492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:35.742370 32389 raft_consensus.cc:1273] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Refusing update from remote peer cb3d5d42c47546258deed617114e71f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 02:04:35.743204 32607 consensus_queue.cc:1035] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:35.743901 32531 tablet_bootstrap.cc:492] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:35.743974 32531 ts_tablet_manager.cc:1397] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20250628 02:04:35.744351 32531 raft_consensus.cc:357] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.744459 32531 raft_consensus.cc:383] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.744490 32531 raft_consensus.cc:738] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.744596 32531 consensus_queue.cc:260] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.744864 32531 ts_tablet_manager.cc:1428] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250628 02:04:35.746228 32530 tablet_bootstrap.cc:492] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:35.746294 32530 ts_tablet_manager.cc:1397] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20250628 02:04:35.746675 32530 raft_consensus.cc:357] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.746757 32530 raft_consensus.cc:383] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.746786 32530 raft_consensus.cc:738] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.746886 32530 consensus_queue.cc:260] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.747110 32530 ts_tablet_manager.cc:1428] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250628 02:04:35.747901 31924 catalog_manager.cc:5674] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733) does not have the latest schema for tablet ba0c39c26dc24ee298e9790092f461cf (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]). Expected version 1 got 0
I20250628 02:04:35.748850 32639 mvcc.cc:204] Tried to move back new op lower bound from 7172408425222889472 to 7172408421753577472. Current Snapshot: MvccSnapshot[applied={T|T < 7172408425179447296}]
I20250628 02:04:35.750084 32532 tablet_bootstrap.cc:492] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:35.750154 32532 ts_tablet_manager.cc:1397] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250628 02:04:35.750470 32532 raft_consensus.cc:357] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.750531 32532 raft_consensus.cc:383] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.750550 32532 raft_consensus.cc:738] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.750648 32532 consensus_queue.cc:260] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.750850 32532 ts_tablet_manager.cc:1428] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250628 02:04:35.753648 32253 raft_consensus.cc:1273] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Refusing update from remote peer cb3d5d42c47546258deed617114e71f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 02:04:35.754354 32607 consensus_queue.cc:1035] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:35.755245 31779 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=56f325f0d2044b03bda5136a7f717212]
I20250628 02:04:35.755429 31779 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51330 (request call id 6) took 1583 ms. Trace:
I20250628 02:04:35.755473 31779 rpcz_store.cc:276] 0628 02:04:34.171880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0628 02:04:34.171919 (+ 39us) service_pool.cc:225] Handling call
0628 02:04:34.210116 (+ 38197us) catalog_manager.cc:2076] Acquired catalog manager lock
0628 02:04:34.210392 (+ 276us) catalog_manager.cc:2122] Created new table and tablet info
0628 02:04:34.740307 (+529915us) catalog_manager.cc:2148] Created new table in HMS catalog
0628 02:04:34.740700 (+ 393us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.740702 (+ 2us) write_op.cc:454] Released schema lock
0628 02:04:34.740825 (+ 123us) catalog_manager.cc:2156] Rolling back HMS table creation
0628 02:04:35.755417 (+1014592us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0628 02:04:34.740663 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER)
0628 02:04:34.740669 (+ 6us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.740673 (+ 4us) write_op.cc:454] Released schema lock
0628 02:04:34.740676 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1450102,"Hive Metastore.run_cpu_time_us":357,"Hive Metastore.run_wall_time_us":93934,"spinlock_wait_cycles":20480,"child_traces":[["op",{}]]}
I20250628 02:04:35.758369 32655 tablet.cc:1722] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250628 02:04:35.759406 31924 catalog_manager.cc:5674] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697) does not have the latest schema for tablet ba0c39c26dc24ee298e9790092f461cf (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]). Expected version 1 got 0
I20250628 02:04:35.760432 32642 mvcc.cc:204] Tried to move back new op lower bound from 7172408425432403968 to 7172408425161752576. Current Snapshot: MvccSnapshot[applied={T|T < 7172408425432403968}]
I20250628 02:04:35.761505 32656 tablet.cc:1722] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250628 02:04:35.763435 31922 catalog_manager.cc:5674] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733) does not have the latest schema for tablet ba0c39c26dc24ee298e9790092f461cf (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]). Expected version 1 got 0
I20250628 02:04:35.763661 31924 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52608:
name: "default.Table_0ba2136e171748f0ac70f94ad2378a15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:35.763972 32650 mvcc.cc:204] Tried to move back new op lower bound from 7172408425432403968 to 7172408425161752576. Current Snapshot: MvccSnapshot[applied={T|T < 7172408425432403968}]
I20250628 02:04:35.770076 31922 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e] alter complete (version 1)
I20250628 02:04:35.770586 31922 catalog_manager.cc:5674] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697) does not have the latest schema for tablet ba0c39c26dc24ee298e9790092f461cf (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]). Expected version 1 got 0
I20250628 02:04:35.771158 32658 tablet.cc:1722] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250628 02:04:35.782001 31922 master_service.cc:948] ReplaceTablet: received request to replace tablet ba0c39c26dc24ee298e9790092f461cf from {username='slave'} at 127.0.0.1:52608
I20250628 02:04:35.783406 32653 mvcc.cc:204] Tried to move back new op lower bound from 7172408425432403968 to 7172408425161752576. Current Snapshot: MvccSnapshot[applied={T|T < 7172408425432403968}]
I20250628 02:04:35.786069 31922 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Sending DeleteTablet for 3 replicas of tablet ba0c39c26dc24ee298e9790092f461cf
I20250628 02:04:35.786510 32099 tablet_service.cc:1515] Processing DeleteTablet for tablet ba0c39c26dc24ee298e9790092f461cf with delete_type TABLET_DATA_DELETED (replaced by tablet b1f806247c6c449c988edf62379b6e11) from {username='slave'} at 127.0.0.1:34860
I20250628 02:04:35.786660 32544 tablet_replica.cc:331] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:35.786757 32544 raft_consensus.cc:2241] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Raft consensus shutting down.
I20250628 02:04:35.786922 32544 raft_consensus.cc:2270] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:35.787505 32544 ts_tablet_manager.cc:1905] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:35.788112 32539 raft_consensus.cc:491] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:35.788210 32539 raft_consensus.cc:513] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.789253 32539 leader_election.cc:290] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:35.789378 32544 ts_tablet_manager.cc:1918] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250628 02:04:35.789407 32234 tablet_service.cc:1515] Processing DeleteTablet for tablet ba0c39c26dc24ee298e9790092f461cf with delete_type TABLET_DATA_DELETED (replaced by tablet b1f806247c6c449c988edf62379b6e11) from {username='slave'} at 127.0.0.1:58162
I20250628 02:04:35.789436 32544 log.cc:1199] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/ba0c39c26dc24ee298e9790092f461cf
I20250628 02:04:35.789661 32544 ts_tablet_manager.cc:1939] T ba0c39c26dc24ee298e9790092f461cf P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:04:35.789808 32543 tablet_replica.cc:331] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:35.789909 32543 raft_consensus.cc:2241] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:35.790066 32543 raft_consensus.cc:2270] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:35.792547 32543 ts_tablet_manager.cc:1905] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:35.794397 32543 ts_tablet_manager.cc:1918] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250628 02:04:35.794452 32543 log.cc:1199] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/ba0c39c26dc24ee298e9790092f461cf
I20250628 02:04:35.794659 32543 ts_tablet_manager.cc:1939] T ba0c39c26dc24ee298e9790092f461cf P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
I20250628 02:04:35.795672 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48ccc4cdd28b422ca19674eb5285230d" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9" is_pre_election: true
I20250628 02:04:35.795872 32119 raft_consensus.cc:2466] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
I20250628 02:04:35.796355 32187 leader_election.cc:304] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:35.796613 31913 catalog_manager.cc:4928] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697): tablet ba0c39c26dc24ee298e9790092f461cf (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]) successfully deleted
I20250628 02:04:35.796792 31910 catalog_manager.cc:4928] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393): tablet ba0c39c26dc24ee298e9790092f461cf (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]) successfully deleted
I20250628 02:04:35.796970 32539 raft_consensus.cc:2802] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:35.797020 32539 raft_consensus.cc:491] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:35.797050 32539 raft_consensus.cc:3058] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:35.797789 32539 raft_consensus.cc:513] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.798089 32539 leader_election.cc:290] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Requested vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:35.798415 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48ccc4cdd28b422ca19674eb5285230d" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9"
I20250628 02:04:35.798527 32119 raft_consensus.cc:3058] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:35.799364 32119 raft_consensus.cc:2466] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
I20250628 02:04:35.799836 32187 leader_election.cc:304] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:35.799865 32369 tablet_service.cc:1515] Processing DeleteTablet for tablet ba0c39c26dc24ee298e9790092f461cf with delete_type TABLET_DATA_DELETED (replaced by tablet b1f806247c6c449c988edf62379b6e11) from {username='slave'} at 127.0.0.1:34424
I20250628 02:04:35.800043 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48ccc4cdd28b422ca19674eb5285230d" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:35.800143 32389 raft_consensus.cc:2466] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
I20250628 02:04:35.800347 32388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48ccc4cdd28b422ca19674eb5285230d" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:35.800433 32388 raft_consensus.cc:3058] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:35.801318 32388 raft_consensus.cc:2466] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
I20250628 02:04:35.801842 32542 tablet_replica.cc:331] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:35.801944 32542 raft_consensus.cc:2241] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:35.802081 32542 raft_consensus.cc:2270] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:35.802830 32539 raft_consensus.cc:2802] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:35.803028 32539 raft_consensus.cc:695] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [term 1 LEADER]: Becoming Leader. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Running, Role: LEADER
I20250628 02:04:35.803040 32542 ts_tablet_manager.cc:1905] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:35.803238 32539 consensus_queue.cc:237] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.806087 32542 ts_tablet_manager.cc:1918] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250628 02:04:35.806058 32234 tablet_service.cc:1468] Processing CreateTablet for tablet b1f806247c6c449c988edf62379b6e11 (DEFAULT_TABLE table=default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.806151 32542 log.cc:1199] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/ba0c39c26dc24ee298e9790092f461cf
I20250628 02:04:35.806286 32234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f806247c6c449c988edf62379b6e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.806386 32542 ts_tablet_manager.cc:1939] T ba0c39c26dc24ee298e9790092f461cf P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:04:35.808025 32531 tablet_bootstrap.cc:492] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:35.808744 32369 tablet_service.cc:1468] Processing CreateTablet for tablet b1f806247c6c449c988edf62379b6e11 (DEFAULT_TABLE table=default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.808890 32531 tablet_bootstrap.cc:654] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.808931 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f806247c6c449c988edf62379b6e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.809989 32099 tablet_service.cc:1468] Processing CreateTablet for tablet b1f806247c6c449c988edf62379b6e11 (DEFAULT_TABLE table=default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.810240 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f806247c6c449c988edf62379b6e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.810753 32530 tablet_bootstrap.cc:492] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:35.811583 32530 tablet_bootstrap.cc:654] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.812160 32532 tablet_bootstrap.cc:492] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:35.813083 32532 tablet_bootstrap.cc:654] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.813273 32531 tablet_bootstrap.cc:492] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:35.813336 32531 ts_tablet_manager.cc:1397] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20250628 02:04:35.813706 31911 catalog_manager.cc:4928] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733): tablet ba0c39c26dc24ee298e9790092f461cf (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]) successfully deleted
I20250628 02:04:35.813709 32531 raft_consensus.cc:357] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.813791 32531 raft_consensus.cc:383] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.813819 32531 raft_consensus.cc:738] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.813917 32531 consensus_queue.cc:260] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.814244 32531 ts_tablet_manager.cc:1428] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:35.815837 31923 catalog_manager.cc:3460] renamed table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc] in HMS: new name default.table_2ffede105ab7483388016f792547fd7f
I20250628 02:04:35.817986 31922 catalog_manager.cc:6652] ReplaceTablet: tablet ba0c39c26dc24ee298e9790092f461cf deleted and replaced by tablet b1f806247c6c449c988edf62379b6e11
I20250628 02:04:35.818840 31922 catalog_manager.cc:5674] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697) does not have the latest schema for tablet b1f806247c6c449c988edf62379b6e11 (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]). Expected version 1 got 0
W20250628 02:04:35.819237 31922 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet b1f806247c6c449c988edf62379b6e11 (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet b1f806247c6c449c988edf62379b6e11 consensus state has no leader
I20250628 02:04:35.819568 31922 master_service.cc:948] ReplaceTablet: received request to replace tablet ba0c39c26dc24ee298e9790092f461cf from {username='slave'} at 127.0.0.1:52608
I20250628 02:04:35.819990 31922 master_service.cc:948] ReplaceTablet: received request to replace tablet ba0c39c26dc24ee298e9790092f461cf from {username='slave'} at 127.0.0.1:52608
I20250628 02:04:35.820797 31922 catalog_manager.cc:5674] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697) does not have the latest schema for tablet b1f806247c6c449c988edf62379b6e11 (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]). Expected version 1 got 0
I20250628 02:04:35.821286 31922 catalog_manager.cc:5582] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2). New cstate: current_term: 1 leader_uuid: "b9085abee1ab4c96a3716383fe8fe592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } health_report { overall_health: UNKNOWN } } }
I20250628 02:04:35.822286 32532 tablet_bootstrap.cc:492] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:35.822367 32532 ts_tablet_manager.cc:1397] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250628 02:04:35.822947 32532 raft_consensus.cc:357] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.823048 32532 raft_consensus.cc:383] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.823081 32532 raft_consensus.cc:738] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.823196 32532 consensus_queue.cc:260] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.823457 32532 ts_tablet_manager.cc:1428] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:35.825860 32530 tablet_bootstrap.cc:492] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:35.825954 32530 ts_tablet_manager.cc:1397] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.001s
I20250628 02:04:35.826351 32530 raft_consensus.cc:357] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.826450 32530 raft_consensus.cc:383] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.826483 32530 raft_consensus.cc:738] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.826578 32530 consensus_queue.cc:260] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.826668 31922 catalog_manager.cc:5674] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393) does not have the latest schema for tablet b1f806247c6c449c988edf62379b6e11 (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]). Expected version 1 got 0
I20250628 02:04:35.826821 32530 ts_tablet_manager.cc:1428] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:35.826982 31922 master_service.cc:948] ReplaceTablet: received request to replace tablet ba0c39c26dc24ee298e9790092f461cf from {username='slave'} at 127.0.0.1:52608
I20250628 02:04:35.827193 31922 master_service.cc:948] ReplaceTablet: received request to replace tablet ba0c39c26dc24ee298e9790092f461cf from {username='slave'} at 127.0.0.1:52608
I20250628 02:04:35.836687 31922 catalog_manager.cc:5674] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733) does not have the latest schema for tablet b1f806247c6c449c988edf62379b6e11 (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]). Expected version 1 got 0
I20250628 02:04:35.838088 31922 catalog_manager.cc:5674] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733) does not have the latest schema for tablet b1f806247c6c449c988edf62379b6e11 (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]). Expected version 1 got 0
I20250628 02:04:35.846597 31918 catalog_manager.cc:2149] created HMS catalog entry for table [id=582401feb808444b885f2532faac36e0]
I20250628 02:04:35.850708 31781 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=92619a7eecd54094bd8c3e908c2314e3]
I20250628 02:04:35.850879 31781 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51330 (request call id 4) took 1680 ms. Trace:
I20250628 02:04:35.850927 31781 rpcz_store.cc:276] 0628 02:04:34.170651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0628 02:04:34.170691 (+ 40us) service_pool.cc:225] Handling call
0628 02:04:34.210296 (+ 39605us) catalog_manager.cc:2076] Acquired catalog manager lock
0628 02:04:34.210545 (+ 249us) catalog_manager.cc:2122] Created new table and tablet info
0628 02:04:34.763712 (+553167us) catalog_manager.cc:2148] Created new table in HMS catalog
0628 02:04:34.764166 (+ 454us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.764167 (+ 1us) write_op.cc:454] Released schema lock
0628 02:04:34.764281 (+ 114us) catalog_manager.cc:2156] Rolling back HMS table creation
0628 02:04:35.850868 (+1086587us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0628 02:04:34.764135 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER)
0628 02:04:34.764140 (+ 5us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.764143 (+ 3us) write_op.cc:454] Released schema lock
0628 02:04:34.764145 (+ 2us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1520483,"Hive Metastore.run_cpu_time_us":334,"Hive Metastore.run_wall_time_us":118820,"spinlock_wait_cycles":22912,"child_traces":[["op",{}]]}
I20250628 02:04:35.852524 31918 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52608:
table { table_name: "default.Table_51acb52242224cffa3ce3ea99517351a" } modify_external_catalogs: true
I20250628 02:04:35.852638 31918 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52608:
table { table_name: "default.Table_51acb52242224cffa3ce3ea99517351a" } modify_external_catalogs: true
I20250628 02:04:35.855531 32369 tablet_service.cc:1468] Processing CreateTablet for tablet e430df7fefd7495b90146ef324778618 (DEFAULT_TABLE table=default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.855558 32099 tablet_service.cc:1468] Processing CreateTablet for tablet e430df7fefd7495b90146ef324778618 (DEFAULT_TABLE table=default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.855829 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e430df7fefd7495b90146ef324778618. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.855832 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e430df7fefd7495b90146ef324778618. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.857690 32532 tablet_bootstrap.cc:492] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:35.857808 32530 tablet_bootstrap.cc:492] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:35.858438 32532 tablet_bootstrap.cc:654] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.858651 32530 tablet_bootstrap.cc:654] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.859277 32234 tablet_service.cc:1468] Processing CreateTablet for tablet e430df7fefd7495b90146ef324778618 (DEFAULT_TABLE table=default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.859527 32234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e430df7fefd7495b90146ef324778618. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.863540 32531 tablet_bootstrap.cc:492] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:35.864660 32531 tablet_bootstrap.cc:654] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.854089 31922 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52608:
name: "default.Table_48b5e32835b14bc489b2a0c3737240a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:35.866204 32532 tablet_bootstrap.cc:492] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:35.866272 32532 ts_tablet_manager.cc:1397] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250628 02:04:35.866628 32532 raft_consensus.cc:357] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.866734 32532 raft_consensus.cc:383] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.866760 32532 raft_consensus.cc:738] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.866858 32532 consensus_queue.cc:260] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.867070 32532 ts_tablet_manager.cc:1428] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:35.870388 32531 tablet_bootstrap.cc:492] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:35.870471 32531 ts_tablet_manager.cc:1397] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250628 02:04:35.870879 32531 raft_consensus.cc:357] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.870963 32531 raft_consensus.cc:383] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.870988 32531 raft_consensus.cc:738] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.871096 32531 consensus_queue.cc:260] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.871353 32531 ts_tablet_manager.cc:1428] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20250628 02:04:35.871961 31910 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet b1f806247c6c449c988edf62379b6e11 (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b1f806247c6c449c988edf62379b6e11 consensus state has no leader
I20250628 02:04:35.872663 32530 tablet_bootstrap.cc:492] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:35.872759 32530 ts_tablet_manager.cc:1397] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20250628 02:04:35.873183 32530 raft_consensus.cc:357] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.873289 32530 raft_consensus.cc:383] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.873319 32530 raft_consensus.cc:738] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.873430 32530 consensus_queue.cc:260] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.873672 32530 ts_tablet_manager.cc:1428] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:35.883354 31919 catalog_manager.cc:2149] created HMS catalog entry for table [id=c726262dad90499b8c9fda125fda1c5c]
I20250628 02:04:35.883747 32540 raft_consensus.cc:491] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:35.883837 32540 raft_consensus.cc:513] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.884941 32540 leader_election.cc:290] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:35.891918 31919 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52608:
table { table_name: "default.Table_d40b0217227a48fc9431fb713c4ed6be" } modify_external_catalogs: true
I20250628 02:04:35.892035 31919 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52608:
table { table_name: "default.Table_d40b0217227a48fc9431fb713c4ed6be" } modify_external_catalogs: true
I20250628 02:04:35.893988 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73d2f50340b14bd08abd193108c40b47" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9" is_pre_election: true
I20250628 02:04:35.894109 32119 raft_consensus.cc:2466] T 73d2f50340b14bd08abd193108c40b47 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 0.
I20250628 02:04:35.894508 32322 leader_election.cc:304] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb3d5d42c47546258deed617114e71f9, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:35.894729 32540 raft_consensus.cc:2802] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:35.894811 32540 raft_consensus.cc:491] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:35.894861 32540 raft_consensus.cc:3058] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:35.895869 32540 raft_consensus.cc:513] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.896237 32540 leader_election.cc:290] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 election: Requested vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:35.897027 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73d2f50340b14bd08abd193108c40b47" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9"
I20250628 02:04:35.897143 32119 raft_consensus.cc:3058] T 73d2f50340b14bd08abd193108c40b47 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:35.897360 32234 tablet_service.cc:1468] Processing CreateTablet for tablet bc144d6967ed41f6b7fcb7c8f8fe3a69 (DEFAULT_TABLE table=default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.897611 32234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc144d6967ed41f6b7fcb7c8f8fe3a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.898000 32119 raft_consensus.cc:2466] T 73d2f50340b14bd08abd193108c40b47 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 1.
I20250628 02:04:35.898463 32369 tablet_service.cc:1468] Processing CreateTablet for tablet bc144d6967ed41f6b7fcb7c8f8fe3a69 (DEFAULT_TABLE table=default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.898685 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc144d6967ed41f6b7fcb7c8f8fe3a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.899482 32531 tablet_bootstrap.cc:492] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:35.900223 32531 tablet_bootstrap.cc:654] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.900460 32530 tablet_bootstrap.cc:492] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:35.897037 32099 tablet_service.cc:1468] Processing CreateTablet for tablet bc144d6967ed41f6b7fcb7c8f8fe3a69 (DEFAULT_TABLE table=default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.900676 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc144d6967ed41f6b7fcb7c8f8fe3a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.901407 32530 tablet_bootstrap.cc:654] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.902374 32531 tablet_bootstrap.cc:492] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:35.902382 32532 tablet_bootstrap.cc:492] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:35.902451 32531 ts_tablet_manager.cc:1397] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250628 02:04:35.902501 32530 tablet_bootstrap.cc:492] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:35.902562 32530 ts_tablet_manager.cc:1397] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250628 02:04:35.902873 32531 raft_consensus.cc:357] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.902913 32530 raft_consensus.cc:357] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.902957 32531 raft_consensus.cc:383] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.902983 32530 raft_consensus.cc:383] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.902984 32531 raft_consensus.cc:738] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.903010 32530 raft_consensus.cc:738] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.903087 32531 consensus_queue.cc:260] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.903088 32530 consensus_queue.cc:260] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.903185 32532 tablet_bootstrap.cc:654] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.903299 32530 ts_tablet_manager.cc:1428] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:35.903298 32531 ts_tablet_manager.cc:1428] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:35.906440 32322 leader_election.cc:304] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb3d5d42c47546258deed617114e71f9, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:35.908139 32532 tablet_bootstrap.cc:492] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:35.908226 32532 ts_tablet_manager.cc:1397] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20250628 02:04:35.908601 32532 raft_consensus.cc:357] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.908685 32532 raft_consensus.cc:383] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.908710 32532 raft_consensus.cc:738] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.908798 32532 consensus_queue.cc:260] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.908998 32532 ts_tablet_manager.cc:1428] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:35.912724 32253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73d2f50340b14bd08abd193108c40b47" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592" is_pre_election: true
I20250628 02:04:35.912827 32253 raft_consensus.cc:2466] T 73d2f50340b14bd08abd193108c40b47 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 0.
I20250628 02:04:35.912994 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73d2f50340b14bd08abd193108c40b47" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592"
I20250628 02:04:35.913064 32254 raft_consensus.cc:3058] T 73d2f50340b14bd08abd193108c40b47 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:35.913903 32254 raft_consensus.cc:2466] T 73d2f50340b14bd08abd193108c40b47 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 1.
I20250628 02:04:35.914423 32540 raft_consensus.cc:2802] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:35.914606 32540 raft_consensus.cc:695] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [term 1 LEADER]: Becoming Leader. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Running, Role: LEADER
I20250628 02:04:35.914856 32540 consensus_queue.cc:237] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.927206 31921 catalog_manager.cc:2149] created HMS catalog entry for table [id=17962a5794864b1583c98a76fe18b003]
I20250628 02:04:35.936941 32234 tablet_service.cc:1468] Processing CreateTablet for tablet 0d3398f6111344a19d8ba2fa5ee5f5dd (DEFAULT_TABLE table=default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.937168 32234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d3398f6111344a19d8ba2fa5ee5f5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.938141 31921 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52608:
name: "default.Table_57afb1ada3224fe1a269b48cdb5e87ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:35.939173 32531 tablet_bootstrap.cc:492] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:35.939329 32099 tablet_service.cc:1468] Processing CreateTablet for tablet 0d3398f6111344a19d8ba2fa5ee5f5dd (DEFAULT_TABLE table=default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.939597 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d3398f6111344a19d8ba2fa5ee5f5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.940073 32531 tablet_bootstrap.cc:654] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.943006 32369 tablet_service.cc:1468] Processing CreateTablet for tablet 0d3398f6111344a19d8ba2fa5ee5f5dd (DEFAULT_TABLE table=default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.943230 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d3398f6111344a19d8ba2fa5ee5f5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.945115 32530 tablet_bootstrap.cc:492] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:35.945725 32532 tablet_bootstrap.cc:492] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:35.946043 32530 tablet_bootstrap.cc:654] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.946889 32532 tablet_bootstrap.cc:654] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.947175 32531 tablet_bootstrap.cc:492] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:35.947265 32531 ts_tablet_manager.cc:1397] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20250628 02:04:35.947669 32531 raft_consensus.cc:357] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.947789 32531 raft_consensus.cc:383] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.947824 32531 raft_consensus.cc:738] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.947930 32531 consensus_queue.cc:260] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.948163 32531 ts_tablet_manager.cc:1428] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250628 02:04:35.948680 31917 catalog_manager.cc:2149] created HMS catalog entry for table [id=72b2e754908e45da8d264fd73e392ffb]
I20250628 02:04:35.948719 31776 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=7e80cddaf6dd4ef1ba6f0b33402f9bbc]
I20250628 02:04:35.948884 31776 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51330 (request call id 9) took 1775 ms. Trace:
I20250628 02:04:35.948930 31776 rpcz_store.cc:276] 0628 02:04:34.173170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0628 02:04:34.173207 (+ 37us) service_pool.cc:225] Handling call
0628 02:04:34.209423 (+ 36216us) catalog_manager.cc:2076] Acquired catalog manager lock
0628 02:04:34.209724 (+ 301us) catalog_manager.cc:2122] Created new table and tablet info
0628 02:04:34.786807 (+577083us) catalog_manager.cc:2148] Created new table in HMS catalog
0628 02:04:34.787282 (+ 475us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.787284 (+ 2us) write_op.cc:454] Released schema lock
0628 02:04:34.787405 (+ 121us) catalog_manager.cc:2156] Rolling back HMS table creation
0628 02:04:35.948871 (+1161466us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0628 02:04:34.787240 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER)
0628 02:04:34.787247 (+ 7us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.787251 (+ 4us) write_op.cc:454] Released schema lock
0628 02:04:34.787254 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1617071,"Hive Metastore.run_cpu_time_us":359,"Hive Metastore.run_wall_time_us":120567,"mutex_wait_us":868,"spinlock_wait_cycles":2688,"child_traces":[["op",{}]]}
F20250628 02:04:35.955255 31983 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 5 out of 9 received events: NotificationEvent(eventId=20, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_ccdc9aecb2d34778937160fc3d5cb4cc, message={"server":"","servicePrincipal":"","db":"default","table":"table_ccdc9aecb2d34778937160fc3d5cb4cc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ccdc9aecb2d34778937160fc3d5cb4cc\",\"kudu.table_id\":\"ae7629d68a7d4fdd9ed00f4b1b046cd3\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_0ba2136e171748f0ac70f94ad2378a15, message={"server":"","servicePrincipal":"","db":"default","table":"table_0ba2136e171748f0ac70f94ad2378a15","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0ba2136e171748f0ac70f94ad2378a15\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_0ba2136e171748f0ac70f94ad2378a15\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0ba2136e171748f0ac70f94ad2378a15\",\"kudu.table_id\":\"56f325f0d2044b03bda5136a7f717212\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1751076275, eventType=ALTER_TABLE, dbName=default, tableName=table_2ffede105ab7483388016f792547fd7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_25f7e6e53154401aa745623d0a9238e5","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_25f7e6e53154401aa745623d0a9238e5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_25f7e6e53154401aa745623d0a9238e5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_25f7e6e53154401aa745623d0a9238e5\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_2ffede105ab7483388016f792547fd7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_2ffede105ab7483388016f792547fd7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2ffede105ab7483388016f792547fd7f\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_48b5e32835b14bc489b2a0c3737240a3, message={"server":"","servicePrincipal":"","db":"default","table":"table_48b5e32835b14bc489b2a0c3737240a3","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_48b5e32835b14bc489b2a0c3737240a3\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_48b5e32835b14bc489b2a0c3737240a3\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_48b5e32835b14bc489b2a0c3737240a3\",\"kudu.table_id\":\"92619a7eecd54094bd8c3e908c2314e3\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_51acb52242224cffa3ce3ea99517351a, message={"server":"","servicePrincipal":"","db":"default","table":"table_51acb52242224cffa3ce3ea99517351a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_51acb52242224cffa3ce3ea99517351a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_51acb52242224cffa3ce3ea99517351a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_51acb52242224cffa3ce3ea99517351a\",\"kudu.table_id\":\"582401feb808444b885f2532faac36e0\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250628 02:04:35.956249 32532 tablet_bootstrap.cc:492] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:35.956332 32532 ts_tablet_manager.cc:1397] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250628 02:04:35.956730 32532 raft_consensus.cc:357] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.956827 32532 raft_consensus.cc:383] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.956862 32532 raft_consensus.cc:738] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.956969 32532 consensus_queue.cc:260] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.957206 32532 ts_tablet_manager.cc:1428] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:35.957234 32530 tablet_bootstrap.cc:492] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:35.957310 32530 ts_tablet_manager.cc:1397] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20250628 02:04:35.957702 32530 raft_consensus.cc:357] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.957792 32530 raft_consensus.cc:383] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.957818 32530 raft_consensus.cc:738] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.957906 32530 consensus_queue.cc:260] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.958134 32530 ts_tablet_manager.cc:1428] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:35.959250 31917 catalog_manager.cc:5582] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0aa6633f9b6481bb29b00d602764685 (127.30.198.3). New cstate: current_term: 1 leader_uuid: "d0aa6633f9b6481bb29b00d602764685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } health_report { overall_health: HEALTHY } } }
I20250628 02:04:35.960208 32099 tablet_service.cc:1468] Processing CreateTablet for tablet 347280b05c2c43fe8185766e9f2e73c6 (DEFAULT_TABLE table=default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.960397 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 347280b05c2c43fe8185766e9f2e73c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.960358 32234 tablet_service.cc:1468] Processing CreateTablet for tablet 347280b05c2c43fe8185766e9f2e73c6 (DEFAULT_TABLE table=default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.960546 32234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 347280b05c2c43fe8185766e9f2e73c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.961908 32532 tablet_bootstrap.cc:492] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:35.962775 32532 tablet_bootstrap.cc:654] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.963043 32531 tablet_bootstrap.cc:492] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:35.963651 32369 tablet_service.cc:1468] Processing CreateTablet for tablet 347280b05c2c43fe8185766e9f2e73c6 (DEFAULT_TABLE table=default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:35.963867 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 347280b05c2c43fe8185766e9f2e73c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:35.965286 32530 tablet_bootstrap.cc:492] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:35.966189 32530 tablet_bootstrap.cc:654] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.967118 32530 tablet_bootstrap.cc:492] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:35.967195 32530 ts_tablet_manager.cc:1397] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250628 02:04:35.967765 32530 raft_consensus.cc:357] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.967854 32530 raft_consensus.cc:383] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.967888 32530 raft_consensus.cc:738] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.967988 32530 consensus_queue.cc:260] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.968209 32530 ts_tablet_manager.cc:1428] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:35.969463 32531 tablet_bootstrap.cc:654] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:35.975257 31917 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52608:
table { table_name: "default.Table_48852943c5c64ddfbde3e614d034c8d6" } modify_external_catalogs: true
I20250628 02:04:35.975389 31917 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52608:
table { table_name: "default.Table_48852943c5c64ddfbde3e614d034c8d6" } modify_external_catalogs: true
I20250628 02:04:35.976052 32531 tablet_bootstrap.cc:492] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:35.976120 32531 ts_tablet_manager.cc:1397] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20250628 02:04:35.976152 32532 tablet_bootstrap.cc:492] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:35.976228 32532 ts_tablet_manager.cc:1397] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20250628 02:04:35.976500 32531 raft_consensus.cc:357] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.976586 32531 raft_consensus.cc:383] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.976548 32532 raft_consensus.cc:357] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.976612 32531 raft_consensus.cc:738] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.976620 32532 raft_consensus.cc:383] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:35.976648 32532 raft_consensus.cc:738] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:35.976715 32531 consensus_queue.cc:260] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.976737 32532 consensus_queue.cc:260] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:35.976927 32531 ts_tablet_manager.cc:1428] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250628 02:04:35.976948 32532 ts_tablet_manager.cc:1428] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
@ 0x7f9be1be0dcd google::LogMessage::Fail() at ??:0
@ 0x7f9be1be4b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f9be1be07cc google::LogMessage::Flush() at ??:0
@ 0x7f9be1be0a11 google::LogMessage::~LogMessage() at ??:0
I20250628 02:04:35.997562 31778 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=cbce79435b8d4b95bb9db33535727e1a]
I20250628 02:04:35.997717 31778 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51330 (request call id 7) took 1825 ms. Trace:
I20250628 02:04:35.997746 31778 rpcz_store.cc:276] 0628 02:04:34.172091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0628 02:04:34.172126 (+ 35us) service_pool.cc:225] Handling call
0628 02:04:34.210541 (+ 38415us) catalog_manager.cc:2076] Acquired catalog manager lock
0628 02:04:34.210772 (+ 231us) catalog_manager.cc:2122] Created new table and tablet info
0628 02:04:34.803718 (+592946us) catalog_manager.cc:2148] Created new table in HMS catalog
0628 02:04:34.804085 (+ 367us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.804086 (+ 1us) write_op.cc:454] Released schema lock
0628 02:04:34.804179 (+ 93us) catalog_manager.cc:2156] Rolling back HMS table creation
0628 02:04:35.997709 (+1193530us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0628 02:04:34.804057 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e9fd188e41da4ec0b493c144accb0344 is not leader of this config: current role FOLLOWER)
0628 02:04:34.804061 (+ 4us) write_op.cc:489] Releasing partition, row and schema locks
0628 02:04:34.804064 (+ 3us) write_op.cc:454] Released schema lock
0628 02:04:34.804066 (+ 2us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1719907,"Hive Metastore.run_cpu_time_us":353,"Hive Metastore.run_wall_time_us":66205,"spinlock_wait_cycles":16000,"child_traces":[["op",{}]]}
@ 0x7f9be6b79c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f9be6b78ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f9be6b7830b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
@ 0x7f9be6b7da12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5576d594e752 std::function<>::operator()() at ??:0
I20250628 02:04:36.010414 32664 raft_consensus.cc:491] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:36.010510 32664 raft_consensus.cc:513] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.010921 32664 leader_election.cc:290] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:36.011483 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "347280b05c2c43fe8185766e9f2e73c6" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9" is_pre_election: true
I20250628 02:04:36.011468 32388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "347280b05c2c43fe8185766e9f2e73c6" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:36.011574 32119 raft_consensus.cc:2466] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
I20250628 02:04:36.011574 32388 raft_consensus.cc:2466] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
@ 0x7f9be30320d1 kudu::Thread::SuperviseThread() at ??:0
I20250628 02:04:36.011858 32188 leader_election.cc:304] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:36.012161 32664 raft_consensus.cc:2802] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:36.012207 32664 raft_consensus.cc:491] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:36.012250 32664 raft_consensus.cc:3058] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.013283 32664 raft_consensus.cc:513] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.013643 32664 leader_election.cc:290] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Requested vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
@ 0x7f9be38216db start_thread at ??:0
I20250628 02:04:36.014019 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "347280b05c2c43fe8185766e9f2e73c6" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9"
I20250628 02:04:36.014114 32119 raft_consensus.cc:3058] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.015138 32119 raft_consensus.cc:2466] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
@ 0x7f9be0d9771f clone at ??:0
I20250628 02:04:36.015882 32187 leader_election.cc:304] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:36.015866 32388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "347280b05c2c43fe8185766e9f2e73c6" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:36.015977 32388 raft_consensus.cc:3058] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.016022 32664 raft_consensus.cc:2802] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:36.016078 32664 raft_consensus.cc:695] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [term 1 LEADER]: Becoming Leader. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Running, Role: LEADER
I20250628 02:04:36.016185 32664 consensus_queue.cc:237] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.016836 32388 raft_consensus.cc:2466] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
I20250628 02:04:36.026882 32657 raft_consensus.cc:491] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:36.026953 32657 raft_consensus.cc:513] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.027303 32657 leader_election.cc:290] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:36.027539 32388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1f806247c6c449c988edf62379b6e11" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:36.027571 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1f806247c6c449c988edf62379b6e11" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592" is_pre_election: true
I20250628 02:04:36.027619 32388 raft_consensus.cc:2466] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:36.027662 32254 raft_consensus.cc:2466] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:36.027844 32053 leader_election.cc:304] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb3d5d42c47546258deed617114e71f9, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:36.028116 32657 raft_consensus.cc:2802] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:36.028172 32657 raft_consensus.cc:491] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:36.028209 32657 raft_consensus.cc:3058] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.028811 32657 raft_consensus.cc:513] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.029052 32657 leader_election.cc:290] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Requested vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:36.029299 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1f806247c6c449c988edf62379b6e11" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592"
I20250628 02:04:36.029381 32254 raft_consensus.cc:3058] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.029744 32388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1f806247c6c449c988edf62379b6e11" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:36.029826 32388 raft_consensus.cc:3058] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.030164 32254 raft_consensus.cc:2466] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:36.030429 32388 raft_consensus.cc:2466] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:36.030480 32055 leader_election.cc:304] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:36.031240 32657 raft_consensus.cc:2802] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:36.031318 32657 raft_consensus.cc:695] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Becoming Leader. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Running, Role: LEADER
I20250628 02:04:36.031424 32657 consensus_queue.cc:237] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.055807 32545 raft_consensus.cc:491] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:36.055895 32545 raft_consensus.cc:513] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.056231 32545 leader_election.cc:290] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:36.056505 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e430df7fefd7495b90146ef324778618" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592" is_pre_election: true
I20250628 02:04:36.056535 32388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e430df7fefd7495b90146ef324778618" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:36.056600 32254 raft_consensus.cc:2466] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:36.056632 32388 raft_consensus.cc:2466] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:36.056859 32053 leader_election.cc:304] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb3d5d42c47546258deed617114e71f9, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:36.057003 32545 raft_consensus.cc:2802] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:36.057062 32545 raft_consensus.cc:491] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:36.057106 32545 raft_consensus.cc:3058] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.057929 32545 raft_consensus.cc:513] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.058247 32545 leader_election.cc:290] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Requested vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:36.058478 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e430df7fefd7495b90146ef324778618" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592"
I20250628 02:04:36.058521 32388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e430df7fefd7495b90146ef324778618" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:36.058571 32254 raft_consensus.cc:3058] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.058595 32388 raft_consensus.cc:3058] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.059288 32388 raft_consensus.cc:2466] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:36.059288 32254 raft_consensus.cc:2466] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:36.059531 32055 leader_election.cc:304] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:36.059655 32545 raft_consensus.cc:2802] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:36.059754 32545 raft_consensus.cc:695] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Becoming Leader. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Running, Role: LEADER
I20250628 02:04:36.059865 32545 consensus_queue.cc:237] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
W20250628 02:04:36.158483 32443 connection.cc:537] client connection to 127.30.198.60:40321 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250628 02:04:36.158887 32443 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.30.198.60:40321)
W20250628 02:04:36.159026 32443 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.30.198.60:40321)
W20250628 02:04:36.159256 32443 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.30.198.60:40321)
I20250628 02:04:36.190491 32673 raft_consensus.cc:491] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:36.190580 32673 raft_consensus.cc:513] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.190893 32673 leader_election.cc:290] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:36.191249 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d3398f6111344a19d8ba2fa5ee5f5dd" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9" is_pre_election: true
I20250628 02:04:36.191286 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d3398f6111344a19d8ba2fa5ee5f5dd" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592" is_pre_election: true
I20250628 02:04:36.191391 32254 raft_consensus.cc:2466] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 0.
I20250628 02:04:36.191391 32119 raft_consensus.cc:2466] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 0.
I20250628 02:04:36.191642 32325 leader_election.cc:304] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:36.191828 32673 raft_consensus.cc:2802] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:36.191875 32673 raft_consensus.cc:491] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:36.191903 32673 raft_consensus.cc:3058] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.192698 32673 raft_consensus.cc:513] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.192983 32673 leader_election.cc:290] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 election: Requested vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:36.193288 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d3398f6111344a19d8ba2fa5ee5f5dd" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592"
I20250628 02:04:36.193310 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d3398f6111344a19d8ba2fa5ee5f5dd" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9"
I20250628 02:04:36.193388 32254 raft_consensus.cc:3058] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.193401 32119 raft_consensus.cc:3058] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.194020 32119 raft_consensus.cc:2466] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 1.
I20250628 02:04:36.194043 32254 raft_consensus.cc:2466] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 1.
I20250628 02:04:36.194273 32322 leader_election.cc:304] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb3d5d42c47546258deed617114e71f9, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:36.194404 32673 raft_consensus.cc:2802] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:36.194482 32673 raft_consensus.cc:695] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [term 1 LEADER]: Becoming Leader. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Running, Role: LEADER
I20250628 02:04:36.194633 32673 consensus_queue.cc:237] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.202560 32664 consensus_queue.cc:1035] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:36.205782 32539 consensus_queue.cc:1035] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:36.273959 32545 raft_consensus.cc:491] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:36.274048 32545 raft_consensus.cc:513] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.274328 32545 leader_election.cc:290] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:36.274659 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc144d6967ed41f6b7fcb7c8f8fe3a69" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592" is_pre_election: true
I20250628 02:04:36.274665 32388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc144d6967ed41f6b7fcb7c8f8fe3a69" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:36.274806 32254 raft_consensus.cc:2466] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:36.274806 32388 raft_consensus.cc:2466] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:36.275059 32055 leader_election.cc:304] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:36.275223 32545 raft_consensus.cc:2802] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:36.275279 32545 raft_consensus.cc:491] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:36.275333 32545 raft_consensus.cc:3058] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.276078 32545 raft_consensus.cc:513] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.276335 32545 leader_election.cc:290] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Requested vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:36.276589 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc144d6967ed41f6b7fcb7c8f8fe3a69" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592"
I20250628 02:04:36.276688 32254 raft_consensus.cc:3058] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.276679 32388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc144d6967ed41f6b7fcb7c8f8fe3a69" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:36.276778 32388 raft_consensus.cc:3058] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.277316 32254 raft_consensus.cc:2466] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:36.277436 32388 raft_consensus.cc:2466] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:36.277685 32055 leader_election.cc:304] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:36.277853 32545 raft_consensus.cc:2802] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:36.277922 32545 raft_consensus.cc:695] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Becoming Leader. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Running, Role: LEADER
I20250628 02:04:36.278035 32545 consensus_queue.cc:237] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.393771 32539 consensus_queue.cc:1035] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:36.397297 32664 consensus_queue.cc:1035] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:36.452589 32545 consensus_queue.cc:1035] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:36.455596 32706 consensus_queue.cc:1035] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:36.462869 32673 consensus_queue.cc:1035] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:36.466110 32540 consensus_queue.cc:1035] T 73d2f50340b14bd08abd193108c40b47 P d0aa6633f9b6481bb29b00d602764685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:36.471704 32710 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:04:36.471797 32710 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:36.472111 32710 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:36.473053 31769 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:36.474578 32715 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:04:36.474692 32715 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:36.475075 32573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "36911adab67246cfbfaf77ef5221fc92" is_pre_election: true
I20250628 02:04:36.475268 32573 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fd188e41da4ec0b493c144accb0344 in term 3.
I20250628 02:04:36.475551 31770 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344; no voters: 492448828d9a404fa245390eba0d56fb
I20250628 02:04:36.475867 32710 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250628 02:04:36.475942 32710 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:04:36.475989 32710 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 3 FOLLOWER]: Advancing to term 4
I20250628 02:04:36.476241 31794 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36911adab67246cfbfaf77ef5221fc92" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "e9fd188e41da4ec0b493c144accb0344" is_pre_election: true
I20250628 02:04:36.476276 32715 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:36.476442 32535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:36.476879 32710 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
W20250628 02:04:36.477067 32535 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:36.477105 31794 raft_consensus.cc:2391] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 36911adab67246cfbfaf77ef5221fc92 in current term 4: Already voted for candidate e9fd188e41da4ec0b493c144accb0344 in this term.
I20250628 02:04:36.477216 32710 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 4 election: Requested vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
I20250628 02:04:36.477442 32534 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:36.477574 32573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "36911adab67246cfbfaf77ef5221fc92"
I20250628 02:04:36.477661 32715 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 3 FOLLOWER]: Advancing to term 4
W20250628 02:04:36.478349 31769 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:36.478392 32715 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250628 02:04:36.479074 32573 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fd188e41da4ec0b493c144accb0344 in term 4.
I20250628 02:04:36.479467 31770 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344; no voters: 492448828d9a404fa245390eba0d56fb
I20250628 02:04:36.479629 32710 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 4 FOLLOWER]: Leader election won for term 4
I20250628 02:04:36.479804 32710 raft_consensus.cc:695] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 4 LEADER]: Becoming Leader. State: Replica: e9fd188e41da4ec0b493c144accb0344, State: Running, Role: LEADER
I20250628 02:04:36.479928 32710 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 3.36, Last appended by leader: 36, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:36.480340 32718 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fd188e41da4ec0b493c144accb0344. Latest consensus state: current_term: 4 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:36.480429 32718 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:36.480641 32719 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 02:04:36.481639 32719 catalog_manager.cc:671] Loaded metadata for table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]
I20250628 02:04:36.481860 32719 catalog_manager.cc:671] Loaded metadata for table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]
I20250628 02:04:36.482038 32719 catalog_manager.cc:671] Loaded metadata for table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]
I20250628 02:04:36.482214 32719 catalog_manager.cc:671] Loaded metadata for table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]
I20250628 02:04:36.482395 32719 catalog_manager.cc:671] Loaded metadata for table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]
I20250628 02:04:36.482565 32719 catalog_manager.cc:671] Loaded metadata for table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]
I20250628 02:04:36.482744 32719 catalog_manager.cc:671] Loaded metadata for table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]
I20250628 02:04:36.483580 32719 tablet_loader.cc:96] loaded metadata for tablet 0d3398f6111344a19d8ba2fa5ee5f5dd (table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003])
I20250628 02:04:36.483788 32719 tablet_loader.cc:96] loaded metadata for tablet 347280b05c2c43fe8185766e9f2e73c6 (table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb])
I20250628 02:04:36.483943 32719 tablet_loader.cc:96] loaded metadata for tablet 48ccc4cdd28b422ca19674eb5285230d (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3])
I20250628 02:04:36.484228 32719 tablet_loader.cc:96] loaded metadata for tablet 73d2f50340b14bd08abd193108c40b47 (table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc])
I20250628 02:04:36.484365 32719 tablet_loader.cc:96] loaded metadata for tablet b1f806247c6c449c988edf62379b6e11 (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e])
I20250628 02:04:36.484571 32719 tablet_loader.cc:96] loaded metadata for tablet bc144d6967ed41f6b7fcb7c8f8fe3a69 (table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c])
I20250628 02:04:36.484716 32719 tablet_loader.cc:96] loaded metadata for tablet e430df7fefd7495b90146ef324778618 (table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0])
I20250628 02:04:36.484824 32719 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 02:04:36.485450 32719 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:36.485494 32719 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 02:04:36.486224 32719 catalog_manager.cc:1506] Loading token signing keys...
I20250628 02:04:36.486582 32719 catalog_manager.cc:5966] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Loaded TSK: 0
I20250628 02:04:36.486939 32719 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 02:04:36.487282 32719 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250628 02:04:36.487601 32719 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 19
I20250628 02:04:36.513749 31778 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_6296f6b4fab44283a0eb5a6ee76c4306"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:36.513792 31781 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_0f9f68e3f7dd46c29954d59183c6b66f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:36.514045 31776 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_bed4f3bffa6d40138ec95e58cc15fba0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:36.514134 31775 master_service.cc:948] ReplaceTablet: received request to replace tablet b1f806247c6c449c988edf62379b6e11 from {username='slave'} at 127.0.0.1:51330
I20250628 02:04:36.514263 31782 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51330:
table { table_name: "default.Table_d40b0217227a48fc9431fb713c4ed6be" } modify_external_catalogs: true
I20250628 02:04:36.514389 31782 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51330:
table { table_name: "default.Table_d40b0217227a48fc9431fb713c4ed6be" } modify_external_catalogs: true
I20250628 02:04:36.514269 31777 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51330:
table { table_name: "default.Table_48852943c5c64ddfbde3e614d034c8d6" } modify_external_catalogs: true
I20250628 02:04:36.514583 31777 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51330:
table { table_name: "default.Table_48852943c5c64ddfbde3e614d034c8d6" } modify_external_catalogs: true
I20250628 02:04:36.514395 31783 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51330:
table { table_name: "default.Table_25f7e6e53154401aa745623d0a9238e5" } new_table_name: "default.Table_2ffede105ab7483388016f792547fd7f" modify_external_catalogs: true
I20250628 02:04:36.514688 31784 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_51147aa1de1d4fac95229bd0283919f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:36.514883 31780 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_4687199de5174487b23a117d7fe19211"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:36.514961 31779 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_7eb9297b9cef4c8e82bbdb793ba152cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:36.516497 32573 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Refusing update from remote peer e9fd188e41da4ec0b493c144accb0344: Log matching property violated. Preceding OpId in replica: term: 3 index: 36. Preceding OpId from leader: term: 4 index: 38. (index mismatch)
I20250628 02:04:36.516875 32718 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
W20250628 02:04:36.518044 31769 consensus_peers.cc:489] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 -> Peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Couldn't send request to peer 492448828d9a404fa245390eba0d56fb. Status: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250628 02:04:36.518261 32715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fd188e41da4ec0b493c144accb0344. Latest consensus state: current_term: 4 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:36.518416 32715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:36.518708 32710 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:36.518805 32710 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:36.519116 32715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:36.519379 32715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
F20250628 02:04:36.519977 32001 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 5 out of 10 received events: NotificationEvent(eventId=20, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_ccdc9aecb2d34778937160fc3d5cb4cc, message={"server":"","servicePrincipal":"","db":"default","table":"table_ccdc9aecb2d34778937160fc3d5cb4cc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ccdc9aecb2d34778937160fc3d5cb4cc\",\"kudu.table_id\":\"ae7629d68a7d4fdd9ed00f4b1b046cd3\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_0ba2136e171748f0ac70f94ad2378a15, message={"server":"","servicePrincipal":"","db":"default","table":"table_0ba2136e171748f0ac70f94ad2378a15","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0ba2136e171748f0ac70f94ad2378a15\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_0ba2136e171748f0ac70f94ad2378a15\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0ba2136e171748f0ac70f94ad2378a15\",\"kudu.table_id\":\"56f325f0d2044b03bda5136a7f717212\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1751076275, eventType=ALTER_TABLE, dbName=default, tableName=table_2ffede105ab7483388016f792547fd7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_25f7e6e53154401aa745623d0a9238e5","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_25f7e6e53154401aa745623d0a9238e5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_25f7e6e53154401aa745623d0a9238e5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_25f7e6e53154401aa745623d0a9238e5\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_2ffede105ab7483388016f792547fd7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_2ffede105ab7483388016f792547fd7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2ffede105ab7483388016f792547fd7f\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_48b5e32835b14bc489b2a0c3737240a3, message={"server":"","servicePrincipal":"","db":"default","table":"table_48b5e32835b14bc489b2a0c3737240a3","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_48b5e32835b14bc489b2a0c3737240a3\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_48b5e32835b14bc489b2a0c3737240a3\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_48b5e32835b14bc489b2a0c3737240a3\",\"kudu.table_id\":\"92619a7eecd54094bd8c3e908c2314e3\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_51acb52242224cffa3ce3ea99517351a, message={"server":"","servicePrincipal":"","db":"default","table":"table_51acb52242224cffa3ce3ea99517351a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_51acb52242224cffa3ce3ea99517351a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_51acb52242224cffa3ce3ea99517351a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_51acb52242224cffa3ce3ea99517351a\",\"kudu.table_id\":\"582401feb808444b885f2532faac36e0\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250628 02:04:36.520080 31775 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Sending DeleteTablet for 3 replicas of tablet b1f806247c6c449c988edf62379b6e11
I20250628 02:04:36.521452 31775 catalog_manager.cc:6652] ReplaceTablet: tablet b1f806247c6c449c988edf62379b6e11 deleted and replaced by tablet bc437af4ee094529ae0078641d819bbc
I20250628 02:04:36.521672 31775 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51330:
name: "default.Table_0ba2136e171748f0ac70f94ad2378a15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fbccae36dcd google::LogMessage::Fail() at ??:0
@ 0x7fbccae3ab93 google::LogMessage::SendToLog() at ??:0
I20250628 02:04:36.526351 32234 tablet_service.cc:1515] Processing DeleteTablet for tablet b1f806247c6c449c988edf62379b6e11 with delete_type TABLET_DATA_DELETED (replaced by tablet bc437af4ee094529ae0078641d819bbc) from {username='slave'} at 127.0.0.1:58166
I20250628 02:04:36.526358 32099 tablet_service.cc:1515] Processing DeleteTablet for tablet b1f806247c6c449c988edf62379b6e11 with delete_type TABLET_DATA_DELETED (replaced by tablet bc437af4ee094529ae0078641d819bbc) from {username='slave'} at 127.0.0.1:34862
I20250628 02:04:36.526695 32728 tablet_replica.cc:331] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:36.526753 32727 tablet_replica.cc:331] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:36.526839 32728 raft_consensus.cc:2241] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:36.526897 32727 raft_consensus.cc:2241] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Raft consensus shutting down.
I20250628 02:04:36.526949 32728 raft_consensus.cc:2270] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7fbccae367cc google::LogMessage::Flush() at ??:0
I20250628 02:04:36.527042 32727 pending_rounds.cc:70] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9: Trying to abort 1 pending ops.
I20250628 02:04:36.527071 32727 pending_rounds.cc:77] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172408426625908736 op_type: NO_OP noop_request { }
I20250628 02:04:36.527127 32727 raft_consensus.cc:2887] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250628 02:04:36.527182 32727 raft_consensus.cc:2270] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:36.527328 32728 ts_tablet_manager.cc:1905] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:36.528606 32727 ts_tablet_manager.cc:1905] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fbccae36a11 google::LogMessage::~LogMessage() at ??:0
I20250628 02:04:36.530766 32727 ts_tablet_manager.cc:1918] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:36.530727 32099 tablet_service.cc:1468] Processing CreateTablet for tablet bc437af4ee094529ae0078641d819bbc (DEFAULT_TABLE table=default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:36.530830 32727 log.cc:1199] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/b1f806247c6c449c988edf62379b6e11
I20250628 02:04:36.530787 32234 tablet_service.cc:1468] Processing CreateTablet for tablet bc437af4ee094529ae0078641d819bbc (DEFAULT_TABLE table=default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:36.530993 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc437af4ee094529ae0078641d819bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:36.531024 32234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc437af4ee094529ae0078641d819bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:36.531066 32727 ts_tablet_manager.cc:1939] T b1f806247c6c449c988edf62379b6e11 P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:04:36.531870 31768 catalog_manager.cc:4928] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393): tablet b1f806247c6c449c988edf62379b6e11 (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]) successfully deleted
@ 0x7fbccfdcfc91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250628 02:04:36.532985 32731 tablet_bootstrap.cc:492] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:36.533537 32732 tablet_bootstrap.cc:492] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:36.533787 32728 ts_tablet_manager.cc:1918] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250628 02:04:36.533839 32728 log.cc:1199] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/b1f806247c6c449c988edf62379b6e11
I20250628 02:04:36.534018 32728 ts_tablet_manager.cc:1939] T b1f806247c6c449c988edf62379b6e11 P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
I20250628 02:04:36.534238 32731 tablet_bootstrap.cc:654] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:36.534494 32732 tablet_bootstrap.cc:654] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:36.535539 32732 tablet_bootstrap.cc:492] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:36.535609 32732 ts_tablet_manager.cc:1397] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:36.536084 32732 raft_consensus.cc:357] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.536166 32732 raft_consensus.cc:383] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:36.536201 32732 raft_consensus.cc:738] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
@ 0x7fbccfdceba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250628 02:04:36.536459 31771 catalog_manager.cc:4928] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697): tablet b1f806247c6c449c988edf62379b6e11 (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]) successfully deleted
I20250628 02:04:36.536571 32732 consensus_queue.cc:260] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.536940 32732 ts_tablet_manager.cc:1428] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250628 02:04:36.538396 32369 tablet_service.cc:1515] Processing DeleteTablet for tablet b1f806247c6c449c988edf62379b6e11 with delete_type TABLET_DATA_DELETED (replaced by tablet bc437af4ee094529ae0078641d819bbc) from {username='slave'} at 127.0.0.1:34438
I20250628 02:04:36.538651 32731 tablet_bootstrap.cc:492] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:36.538702 32731 ts_tablet_manager.cc:1397] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250628 02:04:36.538630 32368 tablet_service.cc:1468] Processing CreateTablet for tablet bc437af4ee094529ae0078641d819bbc (DEFAULT_TABLE table=default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:36.538897 32368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc437af4ee094529ae0078641d819bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:36.538967 32731 raft_consensus.cc:357] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.539032 32731 raft_consensus.cc:383] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:36.539064 32731 raft_consensus.cc:738] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
@ 0x7fbccfdce30b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250628 02:04:36.539175 32731 consensus_queue.cc:260] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.539402 32731 ts_tablet_manager.cc:1428] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
@ 0x7fbccfdd3a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250628 02:04:36.541920 32735 tablet_replica.cc:331] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:36.542057 32735 raft_consensus.cc:2241] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:36.542150 32735 raft_consensus.cc:2270] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:36.542428 32735 ts_tablet_manager.cc:1905] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x55c35ff1e752 std::function<>::operator()() at ??:0
I20250628 02:04:36.543825 32735 ts_tablet_manager.cc:1918] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250628 02:04:36.543877 32735 log.cc:1199] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/b1f806247c6c449c988edf62379b6e11
I20250628 02:04:36.544042 32735 ts_tablet_manager.cc:1939] T b1f806247c6c449c988edf62379b6e11 P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:04:36.544538 31769 catalog_manager.cc:4928] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733): tablet b1f806247c6c449c988edf62379b6e11 (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]) successfully deleted
I20250628 02:04:36.544574 32736 tablet_bootstrap.cc:492] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
@ 0x7fbccc2880d1 kudu::Thread::SuperviseThread() at ??:0
I20250628 02:04:36.545255 32736 tablet_bootstrap.cc:654] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:36.546016 32736 tablet_bootstrap.cc:492] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:36.546077 32736 ts_tablet_manager.cc:1397] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250628 02:04:36.546366 32736 raft_consensus.cc:357] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.546427 32736 raft_consensus.cc:383] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:36.546447 32736 raft_consensus.cc:738] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:36.546536 32736 consensus_queue.cc:260] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.546751 32736 ts_tablet_manager.cc:1428] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
@ 0x7fbccca776db start_thread at ??:0
@ 0x7fbcc9fed71f clone at ??:0
I20250628 02:04:36.639663 32706 raft_consensus.cc:491] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:36.639820 32706 raft_consensus.cc:513] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.640252 32706 leader_election.cc:290] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:36.640611 32388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc437af4ee094529ae0078641d819bbc" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:36.640645 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc437af4ee094529ae0078641d819bbc" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592" is_pre_election: true
I20250628 02:04:36.640729 32388 raft_consensus.cc:2466] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:36.640802 32254 raft_consensus.cc:2466] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:36.641000 32053 leader_election.cc:304] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb3d5d42c47546258deed617114e71f9, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:36.641170 32706 raft_consensus.cc:2802] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:36.641232 32706 raft_consensus.cc:491] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:36.641269 32706 raft_consensus.cc:3058] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.642076 32706 raft_consensus.cc:513] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.642350 32706 leader_election.cc:290] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Requested vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:36.642673 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc437af4ee094529ae0078641d819bbc" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592"
I20250628 02:04:36.642784 32254 raft_consensus.cc:3058] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.642791 32388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc437af4ee094529ae0078641d819bbc" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:36.642890 32388 raft_consensus.cc:3058] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:36.643568 32388 raft_consensus.cc:2466] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:36.643627 32254 raft_consensus.cc:2466] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:36.643867 32053 leader_election.cc:304] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb3d5d42c47546258deed617114e71f9, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:36.644019 32706 raft_consensus.cc:2802] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:36.644116 32706 raft_consensus.cc:695] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Becoming Leader. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Running, Role: LEADER
I20250628 02:04:36.644265 32706 consensus_queue.cc:237] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:36.661993 32540 consensus_queue.cc:1035] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:36.662137 32673 consensus_queue.cc:1035] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250628 02:04:36.681494 32442 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.198.62:42671)
W20250628 02:04:36.683881 32659 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:36.681614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.62:42671 (local address 127.30.198.2:34153)
0628 02:04:36.681749 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:36.681760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0628 02:04:36.681833 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0628 02:04:36.683136 (+ 1303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":76}
W20250628 02:04:36.684366 32697 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:36.682706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.62:42671 (local address 127.0.0.1:51360)
0628 02:04:36.682883 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:36.682896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0628 02:04:36.682966 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0628 02:04:36.683717 (+ 751us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":89}
W20250628 02:04:36.684382 32669 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:36.681517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.62:42671 (local address 127.30.198.3:35481)
0628 02:04:36.681656 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:36.681669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0628 02:04:36.681749 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0628 02:04:36.683967 (+ 2218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":79}
I20250628 02:04:36.704499 32706 consensus_queue.cc:1035] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:36.708002 32755 consensus_queue.cc:1035] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:37.032569 32759 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:37.032660 32759 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:37.033039 32759 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:37.033962 32534 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:37.034041 32535 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:37.034152 32535 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:37.034282 32759 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250628 02:04:37.045471 32706 consensus_queue.cc:1035] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:37.049481 32706 consensus_queue.cc:1035] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:37.616636 301 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:37.616734 301 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:37.617033 301 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:37.617952 32534 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:37.618033 32535 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:37.618108 32535 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:37.618223 301 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250628 02:04:37.843915 31512 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu with pid 32500
I20250628 02:04:38.777477 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.61:35059
--webserver_interface=127.30.198.61
--webserver_port=39629
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250628 02:04:38.869239 316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:04:38.869450 316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:04:38.869493 316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:04:38.872072 316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 02:04:38.872114 316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:04:38.872131 316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 02:04:38.872150 316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 02:04:38.875106 316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.61:35059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.198.61
--webserver_port=39629
--never_fsync=true
--heap_profile_path=/tmp/kudu.316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:04:38.875933 316 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:04:38.876770 316 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 02:04:38.881933 321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:38.881966 324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:38.881937 322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:38.882203 316 server_base.cc:1048] running on GCE node
I20250628 02:04:38.882544 316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:04:38.883193 316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:04:38.884327 316 hybrid_clock.cc:648] HybridClock initialized: now 1751076278884306 us; error 31 us; skew 500 ppm
I20250628 02:04:38.888341 316 webserver.cc:469] Webserver started at http://127.30.198.61:39629/ using document root <none> and password file <none>
I20250628 02:04:38.888746 316 fs_manager.cc:362] Metadata directory not provided
I20250628 02:04:38.888790 316 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:04:38.891513 316 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:38.893142 330 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:38.893914 316 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250628 02:04:38.893991 316 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
uuid: "36911adab67246cfbfaf77ef5221fc92"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:38.894274 316 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:04:38.916658 316 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:04:38.917071 316 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:04:38.917181 316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:04:38.922753 316 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.61:35059
I20250628 02:04:38.922772 382 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.61:35059 every 8 connection(s)
I20250628 02:04:38.923624 316 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/info.pb
I20250628 02:04:38.926726 383 sys_catalog.cc:263] Verifying existing consensus state
I20250628 02:04:38.928001 383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Bootstrap starting.
I20250628 02:04:38.932226 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 316
I20250628 02:04:38.934280 383 log.cc:826] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Log is configured to *not* fsync() on all Append() calls
I20250628 02:04:38.954056 383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Bootstrap replayed 1/1 log segments. Stats: ops{read=39 overwritten=0 applied=39 ignored=0} inserts{seen=26 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 02:04:38.954298 383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Bootstrap complete.
I20250628 02:04:38.956477 383 raft_consensus.cc:357] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:38.956728 383 raft_consensus.cc:738] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36911adab67246cfbfaf77ef5221fc92, State: Initialized, Role: FOLLOWER
I20250628 02:04:38.957008 383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 39, Last appended: 4.39, Last appended by leader: 39, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:38.957494 388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:38.957592 388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:38.957788 383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 02:04:38.962425 402 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:38.962522 402 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: loading cluster ID for follower catalog manager: success
I20250628 02:04:38.964659 402 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: acquiring CA information for follower catalog manager: success
I20250628 02:04:38.964223 316 master_runner.cc:186] Error getting master registration for 127.30.198.62:42671: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111),
I20250628 02:04:38.966045 316 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:04:38.966393 402 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250628 02:04:38.998885 32167 heartbeater.cc:344] Connected to a master server at 127.30.198.61:35059
I20250628 02:04:39.000664 343 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cb3d5d42c47546258deed617114e71f9" instance_seqno: 1751076273823804) as {username='slave'} at 127.30.198.1:41611; Asking this server to re-register.
I20250628 02:04:39.001121 32167 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:39.002246 343 ts_manager.cc:194] Registered new tserver with Master: cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:39.055768 32302 heartbeater.cc:344] Connected to a master server at 127.30.198.61:35059
I20250628 02:04:39.056881 343 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" instance_seqno: 1751076273975274) as {username='slave'} at 127.30.198.2:34503; Asking this server to re-register.
I20250628 02:04:39.057231 32302 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:39.057996 343 ts_manager.cc:194] Registered new tserver with Master: b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:39.058897 32437 heartbeater.cc:344] Connected to a master server at 127.30.198.61:35059
I20250628 02:04:39.059628 343 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" instance_seqno: 1751076274129070) as {username='slave'} at 127.30.198.3:48105; Asking this server to re-register.
I20250628 02:04:39.059994 32437 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:39.060707 343 ts_manager.cc:194] Registered new tserver with Master: d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:39.347165 388 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:39.347337 388 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:39.348491 388 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:39.348914 331 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:39.349798 332 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:04:39.349939 331 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
I20250628 02:04:39.350016 331 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:39.350234 388 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250628 02:04:39.923560 417 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:39.923657 417 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:39.924062 417 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:39.924994 331 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:39.925069 332 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:39.925200 332 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:39.925369 417 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250628 02:04:40.564994 433 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:40.565090 433 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:40.565459 433 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:40.566474 331 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:40.566599 332 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:40.566672 332 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:40.566784 433 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250628 02:04:41.653832 449 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:41.653937 449 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:41.654264 449 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:41.655153 331 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:41.655252 332 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:41.655346 332 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:41.655494 449 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250628 02:04:42.143359 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.62:42671
--webserver_interface=127.30.198.62
--webserver_port=40695
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250628 02:04:42.231675 459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:04:42.231840 459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:04:42.231871 459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:04:42.234393 459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 02:04:42.234445 459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:04:42.234468 459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 02:04:42.234488 459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 02:04:42.238073 459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.62:42671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.198.62
--webserver_port=40695
--never_fsync=true
--heap_profile_path=/tmp/kudu.459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:04:42.238830 459 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:04:42.239660 459 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 02:04:42.245031 464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:42.245045 467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:42.245080 459 server_base.cc:1048] running on GCE node
W20250628 02:04:42.245286 465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:42.245780 459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:04:42.246573 459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:04:42.247732 459 hybrid_clock.cc:648] HybridClock initialized: now 1751076282247721 us; error 27 us; skew 500 ppm
I20250628 02:04:42.249145 459 webserver.cc:469] Webserver started at http://127.30.198.62:40695/ using document root <none> and password file <none>
I20250628 02:04:42.249584 459 fs_manager.cc:362] Metadata directory not provided
I20250628 02:04:42.249653 459 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:04:42.252435 459 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:42.253918 473 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:42.254727 459 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:42.254818 459 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
uuid: "e9fd188e41da4ec0b493c144accb0344"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:42.255098 459 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:04:42.270419 459 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:04:42.270915 459 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:04:42.271059 459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:04:42.277030 459 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.62:42671
I20250628 02:04:42.277038 525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.62:42671 every 8 connection(s)
I20250628 02:04:42.277882 459 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/info.pb
I20250628 02:04:42.278084 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 459
I20250628 02:04:42.281273 526 sys_catalog.cc:263] Verifying existing consensus state
I20250628 02:04:42.282717 526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Bootstrap starting.
I20250628 02:04:42.288906 526 log.cc:826] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Log is configured to *not* fsync() on all Append() calls
I20250628 02:04:42.307003 526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Bootstrap replayed 1/1 log segments. Stats: ops{read=39 overwritten=0 applied=39 ignored=0} inserts{seen=26 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 02:04:42.307266 526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Bootstrap complete.
I20250628 02:04:42.309278 526 raft_consensus.cc:357] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:42.309530 526 raft_consensus.cc:738] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fd188e41da4ec0b493c144accb0344, State: Initialized, Role: FOLLOWER
I20250628 02:04:42.309813 526 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 39, Last appended: 4.39, Last appended by leader: 39, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:42.310106 530 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:42.310238 530 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:42.310606 526 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 02:04:42.324628 544 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:42.324676 544 catalog_manager.cc:1554] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: loading cluster ID for follower catalog manager: success
I20250628 02:04:42.326333 544 catalog_manager.cc:1576] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: acquiring CA information for follower catalog manager: success
I20250628 02:04:42.327399 544 catalog_manager.cc:1604] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250628 02:04:42.336390 459 master_runner.cc:186] Error getting master registration for 127.30.198.60:40321: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111),
I20250628 02:04:42.337702 459 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:04:42.503566 32166 heartbeater.cc:344] Connected to a master server at 127.30.198.62:42671
I20250628 02:04:42.509931 490 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cb3d5d42c47546258deed617114e71f9" instance_seqno: 1751076273823804) as {username='slave'} at 127.30.198.1:49045; Asking this server to re-register.
I20250628 02:04:42.510397 32166 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:42.511372 490 ts_manager.cc:194] Registered new tserver with Master: cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:42.560968 32301 heartbeater.cc:344] Connected to a master server at 127.30.198.62:42671
I20250628 02:04:42.561914 490 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" instance_seqno: 1751076273975274) as {username='slave'} at 127.30.198.2:45721; Asking this server to re-register.
I20250628 02:04:42.562424 32436 heartbeater.cc:344] Connected to a master server at 127.30.198.62:42671
I20250628 02:04:42.562446 32301 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:42.563232 490 ts_manager.cc:194] Registered new tserver with Master: b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:42.563305 489 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" instance_seqno: 1751076274129070) as {username='slave'} at 127.30.198.3:46913; Asking this server to re-register.
I20250628 02:04:42.563666 32436 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:42.564345 489 ts_manager.cc:194] Registered new tserver with Master: d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:42.705516 556 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:42.705622 556 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:42.706048 556 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
I20250628 02:04:42.706883 530 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:42.707059 530 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
W20250628 02:04:42.707252 332 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:42.708473 530 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:42.708887 475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:42.708859 357 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 5 candidate_status { last_received { term: 4 index: 39 } } ignore_live_leader: false dest_uuid: "36911adab67246cfbfaf77ef5221fc92" is_pre_election: true
I20250628 02:04:42.709084 357 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fd188e41da4ec0b493c144accb0344 in term 4.
I20250628 02:04:42.709430 476 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344; no voters:
I20250628 02:04:42.709656 530 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250628 02:04:42.709702 475 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:42.709743 530 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:42.709800 530 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 4 FOLLOWER]: Advancing to term 5
I20250628 02:04:42.710343 500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36911adab67246cfbfaf77ef5221fc92" candidate_term: 5 candidate_status { last_received { term: 4 index: 39 } } ignore_live_leader: false dest_uuid: "e9fd188e41da4ec0b493c144accb0344" is_pre_election: true
I20250628 02:04:42.711593 530 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:42.711854 500 raft_consensus.cc:2391] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 36911adab67246cfbfaf77ef5221fc92 in current term 5: Already voted for candidate e9fd188e41da4ec0b493c144accb0344 in this term.
I20250628 02:04:42.711947 530 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 5 election: Requested vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
I20250628 02:04:42.712196 357 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 5 candidate_status { last_received { term: 4 index: 39 } } ignore_live_leader: false dest_uuid: "36911adab67246cfbfaf77ef5221fc92"
I20250628 02:04:42.712299 357 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 4 FOLLOWER]: Advancing to term 5
I20250628 02:04:42.712716 331 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
W20250628 02:04:42.713194 475 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:42.713843 357 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fd188e41da4ec0b493c144accb0344 in term 5.
I20250628 02:04:42.713940 556 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250628 02:04:42.714113 476 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344; no voters: 492448828d9a404fa245390eba0d56fb
I20250628 02:04:42.714306 530 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 5 FOLLOWER]: Leader election won for term 5
I20250628 02:04:42.714501 530 raft_consensus.cc:695] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 5 LEADER]: Becoming Leader. State: Replica: e9fd188e41da4ec0b493c144accb0344, State: Running, Role: LEADER
I20250628 02:04:42.714745 530 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 4.39, Last appended by leader: 39, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:42.716138 561 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fd188e41da4ec0b493c144accb0344. Latest consensus state: current_term: 5 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:42.716255 561 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:42.716563 563 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 02:04:42.717880 563 catalog_manager.cc:671] Loaded metadata for table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]
I20250628 02:04:42.718127 563 catalog_manager.cc:671] Loaded metadata for table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]
I20250628 02:04:42.718323 563 catalog_manager.cc:671] Loaded metadata for table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]
I20250628 02:04:42.718529 563 catalog_manager.cc:671] Loaded metadata for table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]
I20250628 02:04:42.718709 563 catalog_manager.cc:671] Loaded metadata for table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]
I20250628 02:04:42.718887 563 catalog_manager.cc:671] Loaded metadata for table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]
I20250628 02:04:42.719071 563 catalog_manager.cc:671] Loaded metadata for table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]
I20250628 02:04:42.720028 563 tablet_loader.cc:96] loaded metadata for tablet 0d3398f6111344a19d8ba2fa5ee5f5dd (table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003])
I20250628 02:04:42.720209 563 tablet_loader.cc:96] loaded metadata for tablet 347280b05c2c43fe8185766e9f2e73c6 (table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb])
I20250628 02:04:42.720368 563 tablet_loader.cc:96] loaded metadata for tablet 48ccc4cdd28b422ca19674eb5285230d (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3])
I20250628 02:04:42.720670 563 tablet_loader.cc:96] loaded metadata for tablet 73d2f50340b14bd08abd193108c40b47 (table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc])
I20250628 02:04:42.720971 563 tablet_loader.cc:96] loaded metadata for tablet bc144d6967ed41f6b7fcb7c8f8fe3a69 (table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c])
I20250628 02:04:42.721101 563 tablet_loader.cc:96] loaded metadata for tablet bc437af4ee094529ae0078641d819bbc (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e])
I20250628 02:04:42.721223 563 tablet_loader.cc:96] loaded metadata for tablet e430df7fefd7495b90146ef324778618 (table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0])
I20250628 02:04:42.721293 563 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 02:04:42.721642 563 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:42.721680 563 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 02:04:42.722119 563 catalog_manager.cc:1506] Loading token signing keys...
I20250628 02:04:42.722478 563 catalog_manager.cc:5966] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Loaded TSK: 0
I20250628 02:04:42.723008 563 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 02:04:42.723482 563 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250628 02:04:42.723948 563 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 19
I20250628 02:04:42.790016 489 master_service.cc:948] ReplaceTablet: received request to replace tablet 0d3398f6111344a19d8ba2fa5ee5f5dd from {username='slave'} at 127.0.0.1:50870
I20250628 02:04:42.790328 490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50870:
name: "default.Table_57afb1ada3224fe1a269b48cdb5e87ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:42.790601 483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50870:
name: "default.Table_0f9f68e3f7dd46c29954d59183c6b66f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:42.790890 482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50870:
name: "default.Table_bed4f3bffa6d40138ec95e58cc15fba0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:42.791200 488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50870:
name: "default.Table_48b5e32835b14bc489b2a0c3737240a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:42.791273 481 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50870:
name: "default.Table_4687199de5174487b23a117d7fe19211"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:42.791494 484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50870:
name: "default.Table_6296f6b4fab44283a0eb5a6ee76c4306"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:42.791718 487 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50870:
table { table_name: "default.Table_51acb52242224cffa3ce3ea99517351a" } modify_external_catalogs: true
I20250628 02:04:42.791828 485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50870:
table { table_name: "default.Table_ccdc9aecb2d34778937160fc3d5cb4cc" } modify_external_catalogs: true
I20250628 02:04:42.791858 487 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50870:
table { table_name: "default.Table_51acb52242224cffa3ce3ea99517351a" } modify_external_catalogs: true
I20250628 02:04:42.791899 485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50870:
table { table_name: "default.Table_ccdc9aecb2d34778937160fc3d5cb4cc" } modify_external_catalogs: true
I20250628 02:04:42.792007 486 master_service.cc:948] ReplaceTablet: received request to replace tablet bc144d6967ed41f6b7fcb7c8f8fe3a69 from {username='slave'} at 127.0.0.1:50870
I20250628 02:04:42.794111 357 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Refusing update from remote peer e9fd188e41da4ec0b493c144accb0344: Log matching property violated. Preceding OpId in replica: term: 4 index: 39. Preceding OpId from leader: term: 5 index: 41. (index mismatch)
I20250628 02:04:42.794710 561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
F20250628 02:04:42.796831 543 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 5 out of 10 received events: NotificationEvent(eventId=20, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_ccdc9aecb2d34778937160fc3d5cb4cc, message={"server":"","servicePrincipal":"","db":"default","table":"table_ccdc9aecb2d34778937160fc3d5cb4cc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ccdc9aecb2d34778937160fc3d5cb4cc\",\"kudu.table_id\":\"ae7629d68a7d4fdd9ed00f4b1b046cd3\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_0ba2136e171748f0ac70f94ad2378a15, message={"server":"","servicePrincipal":"","db":"default","table":"table_0ba2136e171748f0ac70f94ad2378a15","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0ba2136e171748f0ac70f94ad2378a15\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_0ba2136e171748f0ac70f94ad2378a15\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0ba2136e171748f0ac70f94ad2378a15\",\"kudu.table_id\":\"56f325f0d2044b03bda5136a7f717212\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1751076275, eventType=ALTER_TABLE, dbName=default, tableName=table_2ffede105ab7483388016f792547fd7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_25f7e6e53154401aa745623d0a9238e5","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_25f7e6e53154401aa745623d0a9238e5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_25f7e6e53154401aa745623d0a9238e5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_25f7e6e53154401aa745623d0a9238e5\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_2ffede105ab7483388016f792547fd7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_2ffede105ab7483388016f792547fd7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2ffede105ab7483388016f792547fd7f\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_48b5e32835b14bc489b2a0c3737240a3, message={"server":"","servicePrincipal":"","db":"default","table":"table_48b5e32835b14bc489b2a0c3737240a3","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_48b5e32835b14bc489b2a0c3737240a3\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_48b5e32835b14bc489b2a0c3737240a3\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_48b5e32835b14bc489b2a0c3737240a3\",\"kudu.table_id\":\"92619a7eecd54094bd8c3e908c2314e3\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_51acb52242224cffa3ce3ea99517351a, message={"server":"","servicePrincipal":"","db":"default","table":"table_51acb52242224cffa3ce3ea99517351a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_51acb52242224cffa3ce3ea99517351a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_51acb52242224cffa3ce3ea99517351a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_51acb52242224cffa3ce3ea99517351a\",\"kudu.table_id\":\"582401feb808444b885f2532faac36e0\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250628 02:04:42.798987 556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fd188e41da4ec0b493c144accb0344. Latest consensus state: current_term: 5 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:42.799109 556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fdc07decdcd google::LogMessage::Fail() at ??:0
W20250628 02:04:42.800539 475 consensus_peers.cc:489] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 -> Peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Couldn't send request to peer 492448828d9a404fa245390eba0d56fb. Status: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250628 02:04:42.800809 530 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:42.801566 556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:42.801653 556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fdc07df0b93 google::LogMessage::SendToLog() at ??:0
I20250628 02:04:42.802738 489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Sending DeleteTablet for 3 replicas of tablet 0d3398f6111344a19d8ba2fa5ee5f5dd
I20250628 02:04:42.800894 530 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:42.803046 486 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Sending DeleteTablet for 3 replicas of tablet bc144d6967ed41f6b7fcb7c8f8fe3a69
I20250628 02:04:42.803095 565 mvcc.cc:204] Tried to move back new op lower bound from 7172408454321172480 to 7172408454001360896. Current Snapshot: MvccSnapshot[applied={T|T < 7172408454319288320}]
I20250628 02:04:42.804009 489 catalog_manager.cc:6652] ReplaceTablet: tablet 0d3398f6111344a19d8ba2fa5ee5f5dd deleted and replaced by tablet d857afe694cf4efb9efc415fe5476691
I20250628 02:04:42.804029 486 catalog_manager.cc:6652] ReplaceTablet: tablet bc144d6967ed41f6b7fcb7c8f8fe3a69 deleted and replaced by tablet 8c79c884094b453b90c79f0873169071
I20250628 02:04:42.804273 489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50870:
name: "default.Table_7eb9297b9cef4c8e82bbdb793ba152cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:42.804347 486 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50870:
table { table_name: "default.Table_48852943c5c64ddfbde3e614d034c8d6" } modify_external_catalogs: true
I20250628 02:04:42.804463 486 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50870:
table { table_name: "default.Table_48852943c5c64ddfbde3e614d034c8d6" } modify_external_catalogs: true
@ 0x7fdc07dec7cc google::LogMessage::Flush() at ??:0
I20250628 02:04:42.809188 32099 tablet_service.cc:1515] Processing DeleteTablet for tablet bc144d6967ed41f6b7fcb7c8f8fe3a69 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c79c884094b453b90c79f0873169071) from {username='slave'} at 127.0.0.1:51650
I20250628 02:04:42.809232 32098 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d3398f6111344a19d8ba2fa5ee5f5dd with delete_type TABLET_DATA_DELETED (replaced by tablet d857afe694cf4efb9efc415fe5476691) from {username='slave'} at 127.0.0.1:51650
@ 0x7fdc07deca11 google::LogMessage::~LogMessage() at ??:0
I20250628 02:04:42.810994 32099 tablet_service.cc:1468] Processing CreateTablet for tablet d857afe694cf4efb9efc415fe5476691 (DEFAULT_TABLE table=default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:42.811089 32098 tablet_service.cc:1468] Processing CreateTablet for tablet 8c79c884094b453b90c79f0873169071 (DEFAULT_TABLE table=default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:42.811196 32369 tablet_service.cc:1515] Processing DeleteTablet for tablet bc144d6967ed41f6b7fcb7c8f8fe3a69 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c79c884094b453b90c79f0873169071) from {username='slave'} at 127.0.0.1:41626
I20250628 02:04:42.811255 32368 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d3398f6111344a19d8ba2fa5ee5f5dd with delete_type TABLET_DATA_DELETED (replaced by tablet d857afe694cf4efb9efc415fe5476691) from {username='slave'} at 127.0.0.1:41626
I20250628 02:04:42.811287 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d857afe694cf4efb9efc415fe5476691. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:42.811534 32367 tablet_service.cc:1468] Processing CreateTablet for tablet d857afe694cf4efb9efc415fe5476691 (DEFAULT_TABLE table=default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:42.811805 32367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d857afe694cf4efb9efc415fe5476691. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:42.811808 32369 tablet_service.cc:1468] Processing CreateTablet for tablet 8c79c884094b453b90c79f0873169071 (DEFAULT_TABLE table=default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:42.812091 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c79c884094b453b90c79f0873169071. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:42.813520 32098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c79c884094b453b90c79f0873169071. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:42.814693 582 tablet_bootstrap.cc:492] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:42.815068 32234 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d3398f6111344a19d8ba2fa5ee5f5dd with delete_type TABLET_DATA_DELETED (replaced by tablet d857afe694cf4efb9efc415fe5476691) from {username='slave'} at 127.0.0.1:54630
I20250628 02:04:42.815340 32233 tablet_service.cc:1515] Processing DeleteTablet for tablet bc144d6967ed41f6b7fcb7c8f8fe3a69 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c79c884094b453b90c79f0873169071) from {username='slave'} at 127.0.0.1:54630
I20250628 02:04:42.815665 580 tablet_replica.cc:331] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:42.815665 32231 tablet_service.cc:1468] Processing CreateTablet for tablet 8c79c884094b453b90c79f0873169071 (DEFAULT_TABLE table=default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:42.815825 580 raft_consensus.cc:2241] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:42.815953 32231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c79c884094b453b90c79f0873169071. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:42.815933 32232 tablet_service.cc:1468] Processing CreateTablet for tablet d857afe694cf4efb9efc415fe5476691 (DEFAULT_TABLE table=default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:42.815979 580 raft_consensus.cc:2270] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:42.816124 32232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d857afe694cf4efb9efc415fe5476691. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:42.816401 580 ts_tablet_manager.cc:1905] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fdc0cd85c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250628 02:04:42.818145 581 tablet_bootstrap.cc:492] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:42.815832 582 tablet_bootstrap.cc:654] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:42.818792 581 tablet_bootstrap.cc:654] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:42.819206 583 tablet_replica.cc:331] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:42.819285 583 raft_consensus.cc:2241] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:42.819351 583 raft_consensus.cc:2270] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:42.819583 583 ts_tablet_manager.cc:1905] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:42.819617 580 ts_tablet_manager.cc:1918] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:42.819677 580 log.cc:1199] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/bc144d6967ed41f6b7fcb7c8f8fe3a69
I20250628 02:04:42.819972 580 ts_tablet_manager.cc:1939] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:04:42.820261 584 tablet_bootstrap.cc:492] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:42.820691 580 tablet_replica.cc:331] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:42.820784 580 raft_consensus.cc:2241] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [term 1 LEADER]: Raft consensus shutting down.
I20250628 02:04:42.820781 475 catalog_manager.cc:4928] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733): tablet bc144d6967ed41f6b7fcb7c8f8fe3a69 (table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]) successfully deleted
I20250628 02:04:42.821024 580 raft_consensus.cc:2270] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:42.821156 583 ts_tablet_manager.cc:1918] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:42.821226 583 log.cc:1199] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/0d3398f6111344a19d8ba2fa5ee5f5dd
I20250628 02:04:42.821348 584 tablet_bootstrap.cc:654] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:42.821363 580 ts_tablet_manager.cc:1905] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:42.821466 583 ts_tablet_manager.cc:1939] T 0d3398f6111344a19d8ba2fa5ee5f5dd P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
@ 0x7fdc0cd84ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250628 02:04:42.821918 583 tablet_replica.cc:331] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:42.822005 583 raft_consensus.cc:2241] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:42.822005 477 catalog_manager.cc:4928] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697): tablet 0d3398f6111344a19d8ba2fa5ee5f5dd (table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]) successfully deleted
I20250628 02:04:42.822090 583 raft_consensus.cc:2270] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:42.822376 583 ts_tablet_manager.cc:1905] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:42.822788 582 tablet_bootstrap.cc:492] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:42.822862 582 ts_tablet_manager.cc:1397] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250628 02:04:42.822954 580 ts_tablet_manager.cc:1918] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:42.823011 580 log.cc:1199] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/0d3398f6111344a19d8ba2fa5ee5f5dd
I20250628 02:04:42.823242 580 ts_tablet_manager.cc:1939] T 0d3398f6111344a19d8ba2fa5ee5f5dd P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:04:42.823482 582 raft_consensus.cc:357] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:42.823586 582 raft_consensus.cc:383] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:42.823622 582 raft_consensus.cc:738] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:42.823840 475 catalog_manager.cc:4928] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733): tablet 0d3398f6111344a19d8ba2fa5ee5f5dd (table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]) successfully deleted
I20250628 02:04:42.823799 582 consensus_queue.cc:260] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:42.823902 579 tablet_replica.cc:331] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:42.823992 579 raft_consensus.cc:2241] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Raft consensus shutting down.
I20250628 02:04:42.824144 579 raft_consensus.cc:2270] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:42.824455 579 ts_tablet_manager.cc:1905] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fdc0cd8430b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250628 02:04:42.825309 582 ts_tablet_manager.cc:1428] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250628 02:04:42.826056 579 ts_tablet_manager.cc:1918] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:42.826102 579 log.cc:1199] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/bc144d6967ed41f6b7fcb7c8f8fe3a69
I20250628 02:04:42.826295 579 ts_tablet_manager.cc:1939] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:04:42.826680 582 tablet_bootstrap.cc:492] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:42.826793 474 catalog_manager.cc:4928] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393): tablet bc144d6967ed41f6b7fcb7c8f8fe3a69 (table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]) successfully deleted
I20250628 02:04:42.826815 579 tablet_replica.cc:331] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:42.826905 579 raft_consensus.cc:2241] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:42.826968 579 raft_consensus.cc:2270] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:42.827194 579 ts_tablet_manager.cc:1905] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:42.827908 582 tablet_bootstrap.cc:654] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
@ 0x7fdc0cd89a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250628 02:04:42.828716 579 ts_tablet_manager.cc:1918] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:42.828768 579 log.cc:1199] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/0d3398f6111344a19d8ba2fa5ee5f5dd
I20250628 02:04:42.828967 579 ts_tablet_manager.cc:1939] T 0d3398f6111344a19d8ba2fa5ee5f5dd P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
@ 0x55f62445c752 std::function<>::operator()() at ??:0
I20250628 02:04:42.829478 474 catalog_manager.cc:4928] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393): tablet 0d3398f6111344a19d8ba2fa5ee5f5dd (table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]) successfully deleted
I20250628 02:04:42.829967 583 ts_tablet_manager.cc:1918] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:42.830024 583 log.cc:1199] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/bc144d6967ed41f6b7fcb7c8f8fe3a69
I20250628 02:04:42.830140 581 tablet_bootstrap.cc:492] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:42.830237 583 ts_tablet_manager.cc:1939] T bc144d6967ed41f6b7fcb7c8f8fe3a69 P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
I20250628 02:04:42.830240 581 ts_tablet_manager.cc:1397] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20250628 02:04:42.830753 477 catalog_manager.cc:4928] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697): tablet bc144d6967ed41f6b7fcb7c8f8fe3a69 (table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]) successfully deleted
I20250628 02:04:42.830873 581 raft_consensus.cc:357] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:42.830981 581 raft_consensus.cc:383] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:42.831019 581 raft_consensus.cc:738] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:42.831151 581 consensus_queue.cc:260] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:42.832382 581 ts_tablet_manager.cc:1428] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
@ 0x7fdc0923e0d1 kudu::Thread::SuperviseThread() at ??:0
I20250628 02:04:42.833782 582 tablet_bootstrap.cc:492] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:42.833843 582 ts_tablet_manager.cc:1397] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250628 02:04:42.834167 582 raft_consensus.cc:357] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:42.834272 582 raft_consensus.cc:383] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:42.834296 582 raft_consensus.cc:738] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:42.834401 582 consensus_queue.cc:260] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:42.834591 582 ts_tablet_manager.cc:1428] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250628 02:04:42.834781 581 tablet_bootstrap.cc:492] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:42.835518 581 tablet_bootstrap.cc:654] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
@ 0x7fdc09a2d6db start_thread at ??:0
I20250628 02:04:42.836316 584 tablet_bootstrap.cc:492] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:42.836382 584 ts_tablet_manager.cc:1397] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250628 02:04:42.836761 584 raft_consensus.cc:357] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:42.836833 584 raft_consensus.cc:383] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:42.836858 584 raft_consensus.cc:738] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:42.836948 584 consensus_queue.cc:260] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:42.837248 584 ts_tablet_manager.cc:1428] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:42.837426 584 tablet_bootstrap.cc:492] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:42.838472 584 tablet_bootstrap.cc:654] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:42.839222 581 tablet_bootstrap.cc:492] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:42.839277 581 ts_tablet_manager.cc:1397] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20250628 02:04:42.839617 581 raft_consensus.cc:357] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:42.839720 581 raft_consensus.cc:383] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:42.839751 581 raft_consensus.cc:738] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:42.839852 581 consensus_queue.cc:260] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
@ 0x7fdc06fa371f clone at ??:0
I20250628 02:04:42.840076 581 ts_tablet_manager.cc:1428] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:42.841794 584 tablet_bootstrap.cc:492] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:42.841866 584 ts_tablet_manager.cc:1397] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250628 02:04:42.842314 584 raft_consensus.cc:357] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:42.842381 584 raft_consensus.cc:383] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:42.842399 584 raft_consensus.cc:738] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:42.842494 584 consensus_queue.cc:260] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:42.842679 584 ts_tablet_manager.cc:1428] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250628 02:04:42.846508 32673 raft_consensus.cc:491] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:42.846599 32673 raft_consensus.cc:513] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:42.846982 32673 leader_election.cc:290] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:42.847214 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d857afe694cf4efb9efc415fe5476691" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592" is_pre_election: true
I20250628 02:04:42.847311 32254 raft_consensus.cc:2466] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 0.
I20250628 02:04:42.847292 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d857afe694cf4efb9efc415fe5476691" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9" is_pre_election: true
I20250628 02:04:42.847375 32119 raft_consensus.cc:2466] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 0.
I20250628 02:04:42.847568 32325 leader_election.cc:304] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:42.847699 32673 raft_consensus.cc:2802] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:42.847740 32673 raft_consensus.cc:491] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:42.847774 32673 raft_consensus.cc:3058] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:42.848366 32673 raft_consensus.cc:513] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:42.848611 32673 leader_election.cc:290] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 election: Requested vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:42.848912 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d857afe694cf4efb9efc415fe5476691" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592"
I20250628 02:04:42.848956 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d857afe694cf4efb9efc415fe5476691" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9"
I20250628 02:04:42.848999 32254 raft_consensus.cc:3058] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:42.849052 32119 raft_consensus.cc:3058] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:42.849757 32254 raft_consensus.cc:2466] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 1.
I20250628 02:04:42.849905 32119 raft_consensus.cc:2466] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 1.
I20250628 02:04:42.850175 32322 leader_election.cc:304] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb3d5d42c47546258deed617114e71f9, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:42.850299 32673 raft_consensus.cc:2802] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:42.850430 32673 raft_consensus.cc:695] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [term 1 LEADER]: Becoming Leader. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Running, Role: LEADER
I20250628 02:04:42.850517 32673 consensus_queue.cc:237] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:42.943009 312 raft_consensus.cc:491] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:42.943142 312 raft_consensus.cc:513] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:42.943598 312 leader_election.cc:290] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:42.944003 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c79c884094b453b90c79f0873169071" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592" is_pre_election: true
I20250628 02:04:42.944129 32254 raft_consensus.cc:2466] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:42.944368 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c79c884094b453b90c79f0873169071" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:42.944418 32055 leader_election.cc:304] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:42.944504 32389 raft_consensus.cc:2466] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:42.944590 312 raft_consensus.cc:2802] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:42.944639 312 raft_consensus.cc:491] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:42.944675 312 raft_consensus.cc:3058] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:42.945519 312 raft_consensus.cc:513] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:42.945850 312 leader_election.cc:290] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Requested vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:42.946141 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c79c884094b453b90c79f0873169071" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592"
I20250628 02:04:42.946244 32254 raft_consensus.cc:3058] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:42.946236 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c79c884094b453b90c79f0873169071" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:42.946341 32389 raft_consensus.cc:3058] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:42.947185 32254 raft_consensus.cc:2466] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:42.947320 32389 raft_consensus.cc:2466] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:42.947439 32055 leader_election.cc:304] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:42.947583 312 raft_consensus.cc:2802] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:42.947773 312 raft_consensus.cc:695] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Becoming Leader. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Running, Role: LEADER
I20250628 02:04:42.947914 312 consensus_queue.cc:237] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
W20250628 02:04:42.978570 32442 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.30.198.62:42671)
W20250628 02:04:42.978818 32442 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.30.198.62:42671)
W20250628 02:04:42.978927 32442 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.30.198.62:42671)
W20250628 02:04:42.979001 32442 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.198.62:42671)
W20250628 02:04:42.980324 32697 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:42.979888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.62:42671 (local address 127.0.0.1:50894)
0628 02:04:42.980198 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:42.980245 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":224}
W20250628 02:04:42.980458 553 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:42.978442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.62:42671 (local address 127.30.198.3:34801)
0628 02:04:42.978606 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:42.978621 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0628 02:04:42.978707 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0628 02:04:42.980333 (+ 1626us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":85}
W20250628 02:04:42.980922 552 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:42.978584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.62:42671 (local address 127.30.198.2:54761)
0628 02:04:42.978731 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:42.978741 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0628 02:04:42.978803 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0628 02:04:42.980836 (+ 2033us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":79}
W20250628 02:04:42.981559 550 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:42.978353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.62:42671 (local address 127.30.198.1:38705)
0628 02:04:42.979569 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:42.979581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0628 02:04:42.979652 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0628 02:04:42.980868 (+ 1216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1147}
I20250628 02:04:43.310382 606 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:43.310487 606 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:43.310843 606 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:43.311722 331 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:43.311930 332 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:43.311993 332 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:43.312130 606 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250628 02:04:43.354053 32673 consensus_queue.cc:1035] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:43.357409 613 consensus_queue.cc:1035] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:43.409226 312 consensus_queue.cc:1035] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:43.412595 620 consensus_queue.cc:1035] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:43.890177 623 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:43.890268 623 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:43.890578 623 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:43.891510 331 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:43.891638 332 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:43.891734 332 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:43.891862 623 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250628 02:04:43.966688 316 master_runner.cc:186] Error getting master registration for 127.30.198.62:42671: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111),
I20250628 02:04:43.966795 316 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:04:44.546023 635 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:44.546150 635 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:44.546527 635 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:44.546989 332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20250628 02:04:44.547660 331 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:44.547758 332 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:44.547834 332 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:44.547966 635 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250628 02:04:45.772454 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.60:40321
--webserver_interface=127.30.198.60
--webserver_port=35681
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250628 02:04:45.863159 648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:04:45.863375 648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:04:45.863435 648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:04:45.867389 648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 02:04:45.867449 648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:04:45.867478 648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 02:04:45.867515 648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 02:04:45.870667 648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.60:40321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.198.60
--webserver_port=35681
--never_fsync=true
--heap_profile_path=/tmp/kudu.648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:04:45.871505 648 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:04:45.872376 648 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250628 02:04:45.875787 659 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:45.875893 659 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:45.876835 659 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:45.877625 331 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:45.877732 332 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:45.877786 332 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:45.877892 659 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250628 02:04:45.878351 655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:45.878466 656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:45.878465 658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:45.878665 648 server_base.cc:1048] running on GCE node
I20250628 02:04:45.879026 648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:04:45.879781 648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:04:45.880916 648 hybrid_clock.cc:648] HybridClock initialized: now 1751076285880909 us; error 21 us; skew 500 ppm
I20250628 02:04:45.885375 648 webserver.cc:469] Webserver started at http://127.30.198.60:35681/ using document root <none> and password file <none>
I20250628 02:04:45.885881 648 fs_manager.cc:362] Metadata directory not provided
I20250628 02:04:45.885949 648 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:04:45.888792 648 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250628 02:04:45.890543 668 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:45.891405 648 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:45.891513 648 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
uuid: "492448828d9a404fa245390eba0d56fb"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:45.891844 648 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:04:45.918830 648 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:04:45.919327 648 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:04:45.919472 648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:04:45.926211 648 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.60:40321
I20250628 02:04:45.926218 723 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.60:40321 every 8 connection(s)
I20250628 02:04:45.927364 648 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/info.pb
I20250628 02:04:45.930514 724 sys_catalog.cc:263] Verifying existing consensus state
I20250628 02:04:45.931833 724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Bootstrap starting.
I20250628 02:04:45.937449 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 648
I20250628 02:04:45.938182 724 log.cc:826] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Log is configured to *not* fsync() on all Append() calls
I20250628 02:04:45.956879 724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=25 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 02:04:45.957191 724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Bootstrap complete.
I20250628 02:04:45.959290 724 raft_consensus.cc:357] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:45.959585 724 raft_consensus.cc:738] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448828d9a404fa245390eba0d56fb, State: Initialized, Role: FOLLOWER
I20250628 02:04:45.959978 724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 3.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:45.960337 728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:45.960440 728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:45.960783 724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: configured and running, proceeding with master startup.
I20250628 02:04:45.966110 742 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:45.966218 742 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: loading cluster ID for follower catalog manager: success
I20250628 02:04:45.966007 648 master_runner.cc:186] Error getting master registration for 127.30.198.62:42671: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111),
I20250628 02:04:45.967883 648 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:04:45.967985 742 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: acquiring CA information for follower catalog manager: success
I20250628 02:04:45.969877 742 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250628 02:04:46.310727 32435 heartbeater.cc:344] Connected to a master server at 127.30.198.60:40321
I20250628 02:04:46.312474 688 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" instance_seqno: 1751076274129070) as {username='slave'} at 127.30.198.3:57845; Asking this server to re-register.
I20250628 02:04:46.312896 32435 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:46.313891 688 ts_manager.cc:194] Registered new tserver with Master: d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:46.361661 32165 heartbeater.cc:344] Connected to a master server at 127.30.198.60:40321
I20250628 02:04:46.362527 688 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cb3d5d42c47546258deed617114e71f9" instance_seqno: 1751076273823804) as {username='slave'} at 127.30.198.1:53109; Asking this server to re-register.
I20250628 02:04:46.362903 32165 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:46.363591 688 ts_manager.cc:194] Registered new tserver with Master: cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:46.421393 32300 heartbeater.cc:344] Connected to a master server at 127.30.198.60:40321
I20250628 02:04:46.422474 688 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" instance_seqno: 1751076273975274) as {username='slave'} at 127.30.198.2:43457; Asking this server to re-register.
I20250628 02:04:46.422894 32300 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:46.423566 688 ts_manager.cc:194] Registered new tserver with Master: b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:46.470947 752 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:46.471115 752 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:46.472362 752 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
W20250628 02:04:46.472790 669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:46.473495 669 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
I20250628 02:04:46.475790 357 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492448828d9a404fa245390eba0d56fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "36911adab67246cfbfaf77ef5221fc92" is_pre_election: true
I20250628 02:04:46.475934 357 raft_consensus.cc:2366] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 492448828d9a404fa245390eba0d56fb for earlier term 4. Current term is 5.
I20250628 02:04:46.476403 671 leader_election.cc:400] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 4 pre-election: Vote denied by peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 492448828d9a404fa245390eba0d56fb for earlier term 4. Current term is 5.
I20250628 02:04:46.476467 671 leader_election.cc:403] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250628 02:04:46.476639 752 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 3 FOLLOWER]: Advancing to term 5
I20250628 02:04:46.478184 752 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 492448828d9a404fa245390eba0d56fb for earlier term 4. Current term is 5.
I20250628 02:04:47.058692 757 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:47.058784 757 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:47.059123 757 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
I20250628 02:04:47.059438 357 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492448828d9a404fa245390eba0d56fb" candidate_term: 6 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "36911adab67246cfbfaf77ef5221fc92" is_pre_election: true
I20250628 02:04:47.059582 357 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 492448828d9a404fa245390eba0d56fb for term 6 because replica has last-logged OpId of term: 5 index: 43, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 36.
W20250628 02:04:47.060043 669 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
I20250628 02:04:47.060094 669 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 492448828d9a404fa245390eba0d56fb; no voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:47.060228 757 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250628 02:04:47.524603 763 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:47.524701 763 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:47.524993 763 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:47.526139 331 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
I20250628 02:04:47.528555 698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36911adab67246cfbfaf77ef5221fc92" candidate_term: 6 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "492448828d9a404fa245390eba0d56fb" is_pre_election: true
I20250628 02:04:47.528791 698 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36911adab67246cfbfaf77ef5221fc92 in term 5.
I20250628 02:04:47.529080 332 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb; no voters: e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:47.529280 763 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250628 02:04:47.529335 763 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:47.529366 763 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 5 FOLLOWER]: Advancing to term 6
I20250628 02:04:47.530084 763 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:47.530457 763 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 election: Requested vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
I20250628 02:04:47.530786 698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36911adab67246cfbfaf77ef5221fc92" candidate_term: 6 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "492448828d9a404fa245390eba0d56fb"
I20250628 02:04:47.530905 698 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 5 FOLLOWER]: Advancing to term 6
W20250628 02:04:47.531353 331 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
I20250628 02:04:47.531963 698 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36911adab67246cfbfaf77ef5221fc92 in term 6.
I20250628 02:04:47.532199 332 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb; no voters: e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:47.532339 763 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 6 FOLLOWER]: Leader election won for term 6
I20250628 02:04:47.532505 763 raft_consensus.cc:695] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 6 LEADER]: Becoming Leader. State: Replica: 36911adab67246cfbfaf77ef5221fc92, State: Running, Role: LEADER
I20250628 02:04:47.532761 763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 5.43, Last appended by leader: 43, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:47.533692 768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36911adab67246cfbfaf77ef5221fc92. Latest consensus state: current_term: 6 leader_uuid: "36911adab67246cfbfaf77ef5221fc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:47.533788 768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:47.534042 770 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 02:04:47.535421 770 catalog_manager.cc:671] Loaded metadata for table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]
I20250628 02:04:47.535727 770 catalog_manager.cc:671] Loaded metadata for table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]
I20250628 02:04:47.535977 770 catalog_manager.cc:671] Loaded metadata for table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]
I20250628 02:04:47.536203 770 catalog_manager.cc:671] Loaded metadata for table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]
I20250628 02:04:47.536420 770 catalog_manager.cc:671] Loaded metadata for table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]
I20250628 02:04:47.536623 770 catalog_manager.cc:671] Loaded metadata for table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]
I20250628 02:04:47.536835 770 catalog_manager.cc:671] Loaded metadata for table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]
I20250628 02:04:47.537912 770 tablet_loader.cc:96] loaded metadata for tablet 347280b05c2c43fe8185766e9f2e73c6 (table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb])
I20250628 02:04:47.538091 770 tablet_loader.cc:96] loaded metadata for tablet 48ccc4cdd28b422ca19674eb5285230d (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3])
I20250628 02:04:47.538390 770 tablet_loader.cc:96] loaded metadata for tablet 73d2f50340b14bd08abd193108c40b47 (table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc])
I20250628 02:04:47.538547 770 tablet_loader.cc:96] loaded metadata for tablet 8c79c884094b453b90c79f0873169071 (table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c])
I20250628 02:04:47.538908 770 tablet_loader.cc:96] loaded metadata for tablet bc437af4ee094529ae0078641d819bbc (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e])
I20250628 02:04:47.539057 770 tablet_loader.cc:96] loaded metadata for tablet d857afe694cf4efb9efc415fe5476691 (table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003])
I20250628 02:04:47.539194 770 tablet_loader.cc:96] loaded metadata for tablet e430df7fefd7495b90146ef324778618 (table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0])
I20250628 02:04:47.539289 770 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 02:04:47.539667 770 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:47.539739 770 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 02:04:47.540227 770 catalog_manager.cc:1506] Loading token signing keys...
I20250628 02:04:47.540622 770 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Loaded TSK: 0
I20250628 02:04:47.541255 770 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 02:04:47.541682 770 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250628 02:04:47.542080 770 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 19
I20250628 02:04:47.602131 347 master_service.cc:948] ReplaceTablet: received request to replace tablet e430df7fefd7495b90146ef324778618 from {username='slave'} at 127.0.0.1:53926
I20250628 02:04:47.602226 343 master_service.cc:948] ReplaceTablet: received request to replace tablet 347280b05c2c43fe8185766e9f2e73c6 from {username='slave'} at 127.0.0.1:53926
I20250628 02:04:47.602521 346 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53926:
table { table_name: "default.Table_aeae2af4c3b941689431955a4b4cacbd" } modify_external_catalogs: true
I20250628 02:04:47.602651 346 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53926:
table { table_name: "default.Table_aeae2af4c3b941689431955a4b4cacbd" } modify_external_catalogs: true
I20250628 02:04:47.602782 339 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53926:
name: "default.Table_48b5e32835b14bc489b2a0c3737240a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:47.603132 342 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53926:
table { table_name: "default.Table_ccdc9aecb2d34778937160fc3d5cb4cc" } modify_external_catalogs: true
I20250628 02:04:47.603220 342 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53926:
table { table_name: "default.Table_ccdc9aecb2d34778937160fc3d5cb4cc" } modify_external_catalogs: true
I20250628 02:04:47.603258 344 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53926:
table { table_name: "default.Table_5f71bbbfb6c04933890499752142ceb3" } new_table_name: "default.Table_4d41a145f10944dca41a0675451382fc" modify_external_catalogs: true
I20250628 02:04:47.603536 341 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53926:
name: "default.Table_f0ea02c8f8f944ebb5b8006e13d1f24c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:47.603750 345 master_service.cc:948] ReplaceTablet: received request to replace tablet bc144d6967ed41f6b7fcb7c8f8fe3a69 from {username='slave'} at 127.0.0.1:53926
I20250628 02:04:47.603852 338 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53926:
name: "default.Table_57afb1ada3224fe1a269b48cdb5e87ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:47.604076 340 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53926:
table { table_name: "default.Table_51acb52242224cffa3ce3ea99517351a" } modify_external_catalogs: true
I20250628 02:04:47.604177 340 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53926:
table { table_name: "default.Table_51acb52242224cffa3ce3ea99517351a" } modify_external_catalogs: true
I20250628 02:04:47.604161 345 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53926:
name: "default.Table_6296f6b4fab44283a0eb5a6ee76c4306"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:47.605461 698 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Refusing update from remote peer 36911adab67246cfbfaf77ef5221fc92: Log matching property violated. Preceding OpId in replica: term: 3 index: 36. Preceding OpId from leader: term: 6 index: 45. (index mismatch)
I20250628 02:04:47.605960 768 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 36, Time since last communication: 0.000s
W20250628 02:04:47.606987 331 consensus_peers.cc:489] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 -> Peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Couldn't send request to peer e9fd188e41da4ec0b493c144accb0344. Status: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20250628 02:04:47.608759 401 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 5 out of 10 received events: NotificationEvent(eventId=20, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_ccdc9aecb2d34778937160fc3d5cb4cc, message={"server":"","servicePrincipal":"","db":"default","table":"table_ccdc9aecb2d34778937160fc3d5cb4cc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ccdc9aecb2d34778937160fc3d5cb4cc\",\"kudu.table_id\":\"ae7629d68a7d4fdd9ed00f4b1b046cd3\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_0ba2136e171748f0ac70f94ad2378a15, message={"server":"","servicePrincipal":"","db":"default","table":"table_0ba2136e171748f0ac70f94ad2378a15","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0ba2136e171748f0ac70f94ad2378a15\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_0ba2136e171748f0ac70f94ad2378a15\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0ba2136e171748f0ac70f94ad2378a15\",\"kudu.table_id\":\"56f325f0d2044b03bda5136a7f717212\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1751076275, eventType=ALTER_TABLE, dbName=default, tableName=table_2ffede105ab7483388016f792547fd7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_25f7e6e53154401aa745623d0a9238e5","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_25f7e6e53154401aa745623d0a9238e5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_25f7e6e53154401aa745623d0a9238e5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_25f7e6e53154401aa745623d0a9238e5\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_2ffede105ab7483388016f792547fd7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_2ffede105ab7483388016f792547fd7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2ffede105ab7483388016f792547fd7f\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_48b5e32835b14bc489b2a0c3737240a3, message={"server":"","servicePrincipal":"","db":"default","table":"table_48b5e32835b14bc489b2a0c3737240a3","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_48b5e32835b14bc489b2a0c3737240a3\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_48b5e32835b14bc489b2a0c3737240a3\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_48b5e32835b14bc489b2a0c3737240a3\",\"kudu.table_id\":\"92619a7eecd54094bd8c3e908c2314e3\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_51acb52242224cffa3ce3ea99517351a, message={"server":"","servicePrincipal":"","db":"default","table":"table_51acb52242224cffa3ce3ea99517351a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_51acb52242224cffa3ce3ea99517351a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_51acb52242224cffa3ce3ea99517351a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_51acb52242224cffa3ce3ea99517351a\",\"kudu.table_id\":\"582401feb808444b885f2532faac36e0\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250628 02:04:47.610846 775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "36911adab67246cfbfaf77ef5221fc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:47.610972 775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:47.611052 777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "36911adab67246cfbfaf77ef5221fc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:47.611215 777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:47.611246 772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36911adab67246cfbfaf77ef5221fc92. Latest consensus state: current_term: 6 leader_uuid: "36911adab67246cfbfaf77ef5221fc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:47.611358 772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f8ffd862dcd google::LogMessage::Fail() at ??:0
I20250628 02:04:47.612516 768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "36911adab67246cfbfaf77ef5221fc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:47.612617 768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:47.613353 772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "36911adab67246cfbfaf77ef5221fc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:47.613426 772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f8ffd866b93 google::LogMessage::SendToLog() at ??:0
I20250628 02:04:47.613649 347 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Sending DeleteTablet for 3 replicas of tablet e430df7fefd7495b90146ef324778618
I20250628 02:04:47.614192 343 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Sending DeleteTablet for 3 replicas of tablet 347280b05c2c43fe8185766e9f2e73c6
I20250628 02:04:47.614450 773 mvcc.cc:204] Tried to move back new op lower bound from 7172408474031484928 to 7172408428462657536. Current Snapshot: MvccSnapshot[applied={T|T < 7172408474027540480}]
I20250628 02:04:47.615123 343 catalog_manager.cc:6652] ReplaceTablet: tablet 347280b05c2c43fe8185766e9f2e73c6 deleted and replaced by tablet e6af82fc366e4121967e834148998420
I20250628 02:04:47.615317 343 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53926:
name: "default.Table_0f9f68e3f7dd46c29954d59183c6b66f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f8ffd8627cc google::LogMessage::Flush() at ??:0
I20250628 02:04:47.615792 347 catalog_manager.cc:6652] ReplaceTablet: tablet e430df7fefd7495b90146ef324778618 deleted and replaced by tablet ac08335565ca46638e022c4588a592f0
I20250628 02:04:47.615986 347 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53926:
name: "default.Table_bed4f3bffa6d40138ec95e58cc15fba0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f8ffd862a11 google::LogMessage::~LogMessage() at ??:0
I20250628 02:04:47.620925 32098 tablet_service.cc:1515] Processing DeleteTablet for tablet 347280b05c2c43fe8185766e9f2e73c6 with delete_type TABLET_DATA_DELETED (replaced by tablet e6af82fc366e4121967e834148998420) from {username='slave'} at 127.0.0.1:51664
I20250628 02:04:47.621132 32099 tablet_service.cc:1515] Processing DeleteTablet for tablet e430df7fefd7495b90146ef324778618 with delete_type TABLET_DATA_DELETED (replaced by tablet ac08335565ca46638e022c4588a592f0) from {username='slave'} at 127.0.0.1:51664
I20250628 02:04:47.621927 32099 tablet_service.cc:1468] Processing CreateTablet for tablet ac08335565ca46638e022c4588a592f0 (DEFAULT_TABLE table=default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:47.622196 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac08335565ca46638e022c4588a592f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:47.623140 32231 tablet_service.cc:1515] Processing DeleteTablet for tablet 347280b05c2c43fe8185766e9f2e73c6 with delete_type TABLET_DATA_DELETED (replaced by tablet e6af82fc366e4121967e834148998420) from {username='slave'} at 127.0.0.1:54644
I20250628 02:04:47.623165 32232 tablet_service.cc:1515] Processing DeleteTablet for tablet e430df7fefd7495b90146ef324778618 with delete_type TABLET_DATA_DELETED (replaced by tablet ac08335565ca46638e022c4588a592f0) from {username='slave'} at 127.0.0.1:54644
I20250628 02:04:47.623382 32234 tablet_service.cc:1468] Processing CreateTablet for tablet e6af82fc366e4121967e834148998420 (DEFAULT_TABLE table=default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:47.623567 32234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6af82fc366e4121967e834148998420. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:47.623927 32233 tablet_service.cc:1468] Processing CreateTablet for tablet ac08335565ca46638e022c4588a592f0 (DEFAULT_TABLE table=default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:47.624126 32233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac08335565ca46638e022c4588a592f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:47.624251 791 tablet_bootstrap.cc:492] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:47.625111 791 tablet_bootstrap.cc:654] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:47.621647 32367 tablet_service.cc:1515] Processing DeleteTablet for tablet 347280b05c2c43fe8185766e9f2e73c6 with delete_type TABLET_DATA_DELETED (replaced by tablet e6af82fc366e4121967e834148998420) from {username='slave'} at 127.0.0.1:41634
I20250628 02:04:47.621706 32369 tablet_service.cc:1515] Processing DeleteTablet for tablet e430df7fefd7495b90146ef324778618 with delete_type TABLET_DATA_DELETED (replaced by tablet ac08335565ca46638e022c4588a592f0) from {username='slave'} at 127.0.0.1:41634
I20250628 02:04:47.625509 32098 tablet_service.cc:1468] Processing CreateTablet for tablet e6af82fc366e4121967e834148998420 (DEFAULT_TABLE table=default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:47.625666 32098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6af82fc366e4121967e834148998420. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:47.625816 790 tablet_replica.cc:331] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:47.625934 790 raft_consensus.cc:2241] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [term 1 LEADER]: Raft consensus shutting down.
I20250628 02:04:47.626217 790 raft_consensus.cc:2270] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:47.626497 790 ts_tablet_manager.cc:1905] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:47.627393 32367 tablet_service.cc:1468] Processing CreateTablet for tablet ac08335565ca46638e022c4588a592f0 (DEFAULT_TABLE table=default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:47.627408 32369 tablet_service.cc:1468] Processing CreateTablet for tablet e6af82fc366e4121967e834148998420 (DEFAULT_TABLE table=default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:47.627535 793 tablet_replica.cc:331] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:47.627629 793 raft_consensus.cc:2241] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:47.627717 32367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac08335565ca46638e022c4588a592f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:47.627753 793 raft_consensus.cc:2270] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:47.627852 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6af82fc366e4121967e834148998420. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:47.628077 793 ts_tablet_manager.cc:1905] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:47.629770 793 ts_tablet_manager.cc:1918] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:47.629827 793 log.cc:1199] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/347280b05c2c43fe8185766e9f2e73c6
I20250628 02:04:47.630137 793 ts_tablet_manager.cc:1939] T 347280b05c2c43fe8185766e9f2e73c6 P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:04:47.630738 332 catalog_manager.cc:4928] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733): tablet 347280b05c2c43fe8185766e9f2e73c6 (table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]) successfully deleted
@ 0x7f90027fbc91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250628 02:04:47.631505 790 ts_tablet_manager.cc:1918] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:47.631546 790 log.cc:1199] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/347280b05c2c43fe8185766e9f2e73c6
I20250628 02:04:47.631742 790 ts_tablet_manager.cc:1939] T 347280b05c2c43fe8185766e9f2e73c6 P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
I20250628 02:04:47.631852 789 tablet_replica.cc:331] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:47.632002 789 raft_consensus.cc:2241] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:47.632103 789 raft_consensus.cc:2270] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:47.632476 789 ts_tablet_manager.cc:1905] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:47.632579 334 catalog_manager.cc:4928] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697): tablet 347280b05c2c43fe8185766e9f2e73c6 (table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]) successfully deleted
I20250628 02:04:47.632764 790 tablet_replica.cc:331] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:47.632853 790 raft_consensus.cc:2241] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:47.632926 790 raft_consensus.cc:2270] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:47.633241 790 ts_tablet_manager.cc:1905] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:47.634215 791 tablet_bootstrap.cc:492] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:47.634294 791 ts_tablet_manager.cc:1397] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
@ 0x7f90027faba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250628 02:04:47.634740 789 ts_tablet_manager.cc:1918] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:47.634804 789 log.cc:1199] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/347280b05c2c43fe8185766e9f2e73c6
I20250628 02:04:47.634800 791 raft_consensus.cc:357] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:47.634912 791 raft_consensus.cc:383] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:47.634953 791 raft_consensus.cc:738] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:47.635156 789 ts_tablet_manager.cc:1939] T 347280b05c2c43fe8185766e9f2e73c6 P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:04:47.635100 791 consensus_queue.cc:260] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:47.635398 791 ts_tablet_manager.cc:1428] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:47.635524 791 tablet_bootstrap.cc:492] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:47.635882 789 tablet_replica.cc:331] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:47.636019 789 raft_consensus.cc:2241] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Raft consensus shutting down.
I20250628 02:04:47.636207 789 raft_consensus.cc:2270] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:47.636222 331 catalog_manager.cc:4928] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393): tablet 347280b05c2c43fe8185766e9f2e73c6 (table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]) successfully deleted
I20250628 02:04:47.636421 791 tablet_bootstrap.cc:654] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:47.636574 789 ts_tablet_manager.cc:1905] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f90027fa30b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250628 02:04:47.638317 790 ts_tablet_manager.cc:1918] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:47.638371 790 log.cc:1199] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/e430df7fefd7495b90146ef324778618
I20250628 02:04:47.638578 790 ts_tablet_manager.cc:1939] T e430df7fefd7495b90146ef324778618 P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
I20250628 02:04:47.639127 334 catalog_manager.cc:4928] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697): tablet e430df7fefd7495b90146ef324778618 (table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]) successfully deleted
I20250628 02:04:47.639326 793 tablet_replica.cc:331] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:47.639434 793 raft_consensus.cc:2241] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:47.639528 793 raft_consensus.cc:2270] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:47.639878 793 ts_tablet_manager.cc:1905] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f90027ffa12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250628 02:04:47.640528 789 ts_tablet_manager.cc:1918] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:47.640594 789 log.cc:1199] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/e430df7fefd7495b90146ef324778618
@ 0x562fc5a4c752 std::function<>::operator()() at ??:0
I20250628 02:04:47.640851 789 ts_tablet_manager.cc:1939] T e430df7fefd7495b90146ef324778618 P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:04:47.641494 331 catalog_manager.cc:4928] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393): tablet e430df7fefd7495b90146ef324778618 (table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]) successfully deleted
I20250628 02:04:47.641842 793 ts_tablet_manager.cc:1918] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:47.641916 793 log.cc:1199] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/e430df7fefd7495b90146ef324778618
I20250628 02:04:47.642134 793 ts_tablet_manager.cc:1939] T e430df7fefd7495b90146ef324778618 P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
@ 0x7f8ffecb40d1 kudu::Thread::SuperviseThread() at ??:0
I20250628 02:04:47.642764 332 catalog_manager.cc:4928] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733): tablet e430df7fefd7495b90146ef324778618 (table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]) successfully deleted
I20250628 02:04:47.643071 791 tablet_bootstrap.cc:492] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:47.643121 791 ts_tablet_manager.cc:1397] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20250628 02:04:47.643489 791 raft_consensus.cc:357] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:47.643590 791 raft_consensus.cc:383] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:47.643623 791 raft_consensus.cc:738] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:47.643761 791 consensus_queue.cc:260] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:47.643981 792 tablet_bootstrap.cc:492] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:47.644007 791 ts_tablet_manager.cc:1428] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
@ 0x7f8fff4a36db start_thread at ??:0
I20250628 02:04:47.644577 795 tablet_bootstrap.cc:492] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:47.645495 795 tablet_bootstrap.cc:654] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
@ 0x7f8ffca1971f clone at ??:0
I20250628 02:04:47.646528 795 tablet_bootstrap.cc:492] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:47.646597 795 ts_tablet_manager.cc:1397] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:47.646940 792 tablet_bootstrap.cc:654] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:47.647184 795 raft_consensus.cc:357] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:47.647269 795 raft_consensus.cc:383] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:47.647294 795 raft_consensus.cc:738] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:47.647410 795 consensus_queue.cc:260] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:47.647667 795 ts_tablet_manager.cc:1428] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:47.647815 795 tablet_bootstrap.cc:492] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:47.648550 792 tablet_bootstrap.cc:492] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:47.648623 792 ts_tablet_manager.cc:1397] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250628 02:04:47.648674 795 tablet_bootstrap.cc:654] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:47.649555 795 tablet_bootstrap.cc:492] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:47.649562 792 raft_consensus.cc:357] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:47.649623 795 ts_tablet_manager.cc:1397] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250628 02:04:47.649652 792 raft_consensus.cc:383] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:47.649682 792 raft_consensus.cc:738] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:47.649839 792 consensus_queue.cc:260] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:47.650018 795 raft_consensus.cc:357] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:47.650101 795 raft_consensus.cc:383] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:47.650147 795 raft_consensus.cc:738] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:47.650223 792 ts_tablet_manager.cc:1428] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:47.650267 795 consensus_queue.cc:260] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:47.650369 792 tablet_bootstrap.cc:492] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:47.650490 795 ts_tablet_manager.cc:1428] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250628 02:04:47.651810 792 tablet_bootstrap.cc:654] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:47.653663 792 tablet_bootstrap.cc:492] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:47.653725 792 ts_tablet_manager.cc:1397] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250628 02:04:47.654006 792 raft_consensus.cc:357] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:47.654088 792 raft_consensus.cc:383] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:47.654115 792 raft_consensus.cc:738] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:47.654212 792 consensus_queue.cc:260] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:47.654484 792 ts_tablet_manager.cc:1428] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250628 02:04:47.673967 620 raft_consensus.cc:491] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:47.674067 620 raft_consensus.cc:513] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:47.674401 620 leader_election.cc:290] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:47.674722 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6af82fc366e4121967e834148998420" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592" is_pre_election: true
I20250628 02:04:47.674723 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6af82fc366e4121967e834148998420" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:47.674837 32389 raft_consensus.cc:2466] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:47.674837 32254 raft_consensus.cc:2466] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:47.675103 32053 leader_election.cc:304] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb3d5d42c47546258deed617114e71f9, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:47.675256 620 raft_consensus.cc:2802] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:47.675323 620 raft_consensus.cc:491] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:47.675349 620 raft_consensus.cc:3058] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:47.676119 620 raft_consensus.cc:513] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:47.676421 620 leader_election.cc:290] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Requested vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:47.676721 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6af82fc366e4121967e834148998420" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:47.676733 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6af82fc366e4121967e834148998420" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592"
I20250628 02:04:47.676831 32254 raft_consensus.cc:3058] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:47.676831 32389 raft_consensus.cc:3058] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:47.677676 32254 raft_consensus.cc:2466] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:47.677676 32389 raft_consensus.cc:2466] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:47.677943 32055 leader_election.cc:304] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:47.678102 620 raft_consensus.cc:2802] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:47.678280 620 raft_consensus.cc:695] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Becoming Leader. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Running, Role: LEADER
I20250628 02:04:47.678426 620 consensus_queue.cc:237] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
W20250628 02:04:47.782480 32189 connection.cc:537] client connection to 127.30.198.61:35059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250628 02:04:47.782666 32324 connection.cc:537] client connection to 127.30.198.61:35059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250628 02:04:47.783921 32444 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.30.198.61:35059)
W20250628 02:04:47.784153 32444 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.198.61:35059)
W20250628 02:04:47.784272 32444 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.30.198.61:35059)
W20250628 02:04:47.784471 32444 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.30.198.61:35059)
W20250628 02:04:47.784865 762 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:47.783034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.61:35059 (local address 127.30.198.2:55717)
0628 02:04:47.783204 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:47.783219 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0628 02:04:47.783319 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0628 02:04:47.784735 (+ 1416us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":85}
W20250628 02:04:47.785109 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:47.783578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.61:35059 (local address 127.30.198.3:36009)
0628 02:04:47.783802 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:47.783815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0628 02:04:47.783903 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0628 02:04:47.784995 (+ 1092us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":142}
W20250628 02:04:47.785137 761 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:47.782990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.61:35059 (local address 127.30.198.1:44159)
0628 02:04:47.783204 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:47.783219 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0628 02:04:47.783319 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0628 02:04:47.785030 (+ 1711us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":118}
I20250628 02:04:48.060019 620 raft_consensus.cc:491] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:48.060122 620 raft_consensus.cc:513] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:48.060436 620 leader_election.cc:290] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:48.060680 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac08335565ca46638e022c4588a592f0" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592" is_pre_election: true
I20250628 02:04:48.060739 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac08335565ca46638e022c4588a592f0" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:48.060794 32254 raft_consensus.cc:2466] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:48.060863 32389 raft_consensus.cc:2466] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 0.
I20250628 02:04:48.061017 32055 leader_election.cc:304] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:48.061172 620 raft_consensus.cc:2802] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:48.061224 620 raft_consensus.cc:491] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:48.061251 620 raft_consensus.cc:3058] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:48.061997 620 raft_consensus.cc:513] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:48.062260 620 leader_election.cc:290] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Requested vote from peers b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:48.062530 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac08335565ca46638e022c4588a592f0" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:48.062523 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac08335565ca46638e022c4588a592f0" candidate_uuid: "cb3d5d42c47546258deed617114e71f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592"
I20250628 02:04:48.062628 32389 raft_consensus.cc:3058] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:48.062651 32254 raft_consensus.cc:3058] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:48.063489 32254 raft_consensus.cc:2466] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:48.063490 32389 raft_consensus.cc:2466] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb3d5d42c47546258deed617114e71f9 in term 1.
I20250628 02:04:48.063786 32055 leader_election.cc:304] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:48.063963 620 raft_consensus.cc:2802] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:48.064044 620 raft_consensus.cc:695] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Becoming Leader. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Running, Role: LEADER
I20250628 02:04:48.064218 620 consensus_queue.cc:237] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:48.122830 817 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 36911adab67246cfbfaf77ef5221fc92)
I20250628 02:04:48.122921 817 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:48.123250 817 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
W20250628 02:04:48.124127 671 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:04:48.124256 669 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
I20250628 02:04:48.124308 669 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 492448828d9a404fa245390eba0d56fb; no voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:48.124413 817 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250628 02:04:48.126958 620 consensus_queue.cc:1035] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:48.129599 812 consensus_queue.cc:1035] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:48.618896 812 consensus_queue.cc:1035] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:48.622582 812 consensus_queue.cc:1035] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:48.792766 832 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 36911adab67246cfbfaf77ef5221fc92)
I20250628 02:04:48.792865 832 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:48.793149 832 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
W20250628 02:04:48.794193 671 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:04:48.794281 669 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
I20250628 02:04:48.794334 669 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 492448828d9a404fa245390eba0d56fb; no voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:48.794445 832 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250628 02:04:49.257201 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.62:42671
--webserver_interface=127.30.198.62
--webserver_port=40695
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250628 02:04:49.346463 841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:04:49.346626 841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:04:49.346663 841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:04:49.349261 841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 02:04:49.349301 841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:04:49.349318 841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 02:04:49.349332 841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 02:04:49.352407 841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.62:42671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.198.62
--webserver_port=40695
--never_fsync=true
--heap_profile_path=/tmp/kudu.841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:04:49.353096 841 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:04:49.353888 841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 02:04:49.359189 849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:49.359195 846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:49.359200 847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:49.359422 841 server_base.cc:1048] running on GCE node
I20250628 02:04:49.359828 841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:04:49.360523 841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:04:49.361652 841 hybrid_clock.cc:648] HybridClock initialized: now 1751076289361642 us; error 28 us; skew 500 ppm
I20250628 02:04:49.363085 841 webserver.cc:469] Webserver started at http://127.30.198.62:40695/ using document root <none> and password file <none>
I20250628 02:04:49.363515 841 fs_manager.cc:362] Metadata directory not provided
I20250628 02:04:49.363567 841 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:04:49.366101 841 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:49.367697 855 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:49.368491 841 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:49.368577 841 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
uuid: "e9fd188e41da4ec0b493c144accb0344"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:49.368875 841 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:04:49.389204 841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:04:49.389652 841 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:04:49.389781 841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:04:49.390252 860 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 36911adab67246cfbfaf77ef5221fc92)
I20250628 02:04:49.390345 860 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:49.390987 860 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
W20250628 02:04:49.391844 671 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:04:49.391985 669 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
I20250628 02:04:49.392066 669 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 492448828d9a404fa245390eba0d56fb; no voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:49.392196 860 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250628 02:04:49.396502 841 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.62:42671
I20250628 02:04:49.396517 911 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.62:42671 every 8 connection(s)
I20250628 02:04:49.397441 841 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/info.pb
I20250628 02:04:49.400311 912 sys_catalog.cc:263] Verifying existing consensus state
I20250628 02:04:49.401551 912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Bootstrap starting.
I20250628 02:04:49.401913 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 841
I20250628 02:04:49.408649 912 log.cc:826] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Log is configured to *not* fsync() on all Append() calls
I20250628 02:04:49.428836 912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Bootstrap replayed 1/1 log segments. Stats: ops{read=43 overwritten=0 applied=43 ignored=0} inserts{seen=28 ignored=0} mutations{seen=27 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 02:04:49.429109 912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Bootstrap complete.
I20250628 02:04:49.431181 912 raft_consensus.cc:357] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:49.431468 912 raft_consensus.cc:738] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fd188e41da4ec0b493c144accb0344, State: Initialized, Role: FOLLOWER
I20250628 02:04:49.431871 912 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 43, Last appended: 5.43, Last appended by leader: 43, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:49.432215 916 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:49.432315 916 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:49.432693 912 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 02:04:49.437951 930 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:49.438047 930 catalog_manager.cc:1554] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: loading cluster ID for follower catalog manager: success
I20250628 02:04:49.440871 930 catalog_manager.cc:1576] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: acquiring CA information for follower catalog manager: success
I20250628 02:04:49.442392 930 catalog_manager.cc:1604] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250628 02:04:49.442577 841 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:04:49.444440 841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:04:49.627533 32436 heartbeater.cc:344] Connected to a master server at 127.30.198.62:42671
I20250628 02:04:49.629452 876 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" instance_seqno: 1751076274129070) as {username='slave'} at 127.30.198.3:33123; Asking this server to re-register.
I20250628 02:04:49.629720 32301 heartbeater.cc:344] Connected to a master server at 127.30.198.62:42671
I20250628 02:04:49.629879 32436 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:49.630676 875 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" instance_seqno: 1751076273975274) as {username='slave'} at 127.30.198.2:59597; Asking this server to re-register.
I20250628 02:04:49.630970 876 ts_manager.cc:194] Registered new tserver with Master: d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:49.631071 32301 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:49.631855 876 ts_manager.cc:194] Registered new tserver with Master: b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:49.665184 32166 heartbeater.cc:344] Connected to a master server at 127.30.198.62:42671
I20250628 02:04:49.666215 875 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cb3d5d42c47546258deed617114e71f9" instance_seqno: 1751076273823804) as {username='slave'} at 127.30.198.1:41837; Asking this server to re-register.
I20250628 02:04:49.666553 32166 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:49.667234 875 ts_manager.cc:194] Registered new tserver with Master: cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:50.041456 943 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:50.041647 943 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:50.042719 943 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:50.043265 858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:04:50.044101 858 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:04:50.045867 698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 6 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "492448828d9a404fa245390eba0d56fb" is_pre_election: true
I20250628 02:04:50.046020 698 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e9fd188e41da4ec0b493c144accb0344 in current term 6: Already voted for candidate 36911adab67246cfbfaf77ef5221fc92 in this term.
I20250628 02:04:50.046403 857 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:04:50.046625 943 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 5 FOLLOWER]: Advancing to term 6
I20250628 02:04:50.048247 943 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250628 02:04:50.669620 951 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:50.669720 951 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:50.670055 951 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
I20250628 02:04:50.670466 698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 7 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "492448828d9a404fa245390eba0d56fb" is_pre_election: true
I20250628 02:04:50.670642 698 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e9fd188e41da4ec0b493c144accb0344 for term 7 because replica has last-logged OpId of term: 6 index: 47, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 43.
W20250628 02:04:50.671164 858 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:04:50.671221 858 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:04:50.671339 951 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250628 02:04:50.724471 954 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 36911adab67246cfbfaf77ef5221fc92)
I20250628 02:04:50.724558 954 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:50.724869 954 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
W20250628 02:04:50.726039 671 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:04:50.728791 886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492448828d9a404fa245390eba0d56fb" candidate_term: 7 candidate_status { last_received { term: 6 index: 47 } } ignore_live_leader: false dest_uuid: "e9fd188e41da4ec0b493c144accb0344" is_pre_election: true
I20250628 02:04:50.729046 886 raft_consensus.cc:2466] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448828d9a404fa245390eba0d56fb in term 6.
I20250628 02:04:50.729362 669 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92
I20250628 02:04:50.729548 954 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250628 02:04:50.729609 954 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 36911adab67246cfbfaf77ef5221fc92)
I20250628 02:04:50.729645 954 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 6 FOLLOWER]: Advancing to term 7
I20250628 02:04:50.730382 954 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:50.730655 954 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 election: Requested vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
I20250628 02:04:50.731019 886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492448828d9a404fa245390eba0d56fb" candidate_term: 7 candidate_status { last_received { term: 6 index: 47 } } ignore_live_leader: false dest_uuid: "e9fd188e41da4ec0b493c144accb0344"
I20250628 02:04:50.731122 886 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 6 FOLLOWER]: Advancing to term 7
W20250628 02:04:50.731591 671 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:04:50.732014 886 raft_consensus.cc:2466] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448828d9a404fa245390eba0d56fb in term 7.
I20250628 02:04:50.732263 669 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92
I20250628 02:04:50.732442 954 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 7 FOLLOWER]: Leader election won for term 7
I20250628 02:04:50.732609 954 raft_consensus.cc:695] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 7 LEADER]: Becoming Leader. State: Replica: 492448828d9a404fa245390eba0d56fb, State: Running, Role: LEADER
I20250628 02:04:50.732872 954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 6.47, Last appended by leader: 47, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:50.733785 959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 492448828d9a404fa245390eba0d56fb. Latest consensus state: current_term: 7 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:50.733959 959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:50.734279 961 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 02:04:50.735752 961 catalog_manager.cc:671] Loaded metadata for table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]
I20250628 02:04:50.736012 961 catalog_manager.cc:671] Loaded metadata for table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]
I20250628 02:04:50.736233 961 catalog_manager.cc:671] Loaded metadata for table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]
I20250628 02:04:50.736469 961 catalog_manager.cc:671] Loaded metadata for table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]
I20250628 02:04:50.736681 961 catalog_manager.cc:671] Loaded metadata for table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]
I20250628 02:04:50.736888 961 catalog_manager.cc:671] Loaded metadata for table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]
I20250628 02:04:50.737102 961 catalog_manager.cc:671] Loaded metadata for table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]
I20250628 02:04:50.738206 961 tablet_loader.cc:96] loaded metadata for tablet 48ccc4cdd28b422ca19674eb5285230d (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3])
I20250628 02:04:50.738512 961 tablet_loader.cc:96] loaded metadata for tablet 73d2f50340b14bd08abd193108c40b47 (table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc])
I20250628 02:04:50.738653 961 tablet_loader.cc:96] loaded metadata for tablet 8c79c884094b453b90c79f0873169071 (table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c])
I20250628 02:04:50.738794 961 tablet_loader.cc:96] loaded metadata for tablet ac08335565ca46638e022c4588a592f0 (table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0])
I20250628 02:04:50.739130 961 tablet_loader.cc:96] loaded metadata for tablet bc437af4ee094529ae0078641d819bbc (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e])
I20250628 02:04:50.739277 961 tablet_loader.cc:96] loaded metadata for tablet d857afe694cf4efb9efc415fe5476691 (table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003])
I20250628 02:04:50.739485 961 tablet_loader.cc:96] loaded metadata for tablet e6af82fc366e4121967e834148998420 (table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb])
I20250628 02:04:50.739580 961 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 02:04:50.739990 961 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:50.740034 961 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 02:04:50.740526 961 catalog_manager.cc:1506] Loading token signing keys...
I20250628 02:04:50.740913 961 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Loaded TSK: 0
I20250628 02:04:50.741508 961 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 02:04:50.741901 961 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250628 02:04:50.742285 961 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 19
I20250628 02:04:50.782212 688 master_service.cc:948] ReplaceTablet: received request to replace tablet 347280b05c2c43fe8185766e9f2e73c6 from {username='slave'} at 127.0.0.1:56478
I20250628 02:04:50.782881 678 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56478:
table { table_name: "default.Table_ccdc9aecb2d34778937160fc3d5cb4cc" } modify_external_catalogs: true
I20250628 02:04:50.782852 686 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56478:
name: "default.Table_0ba2136e171748f0ac70f94ad2378a15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:50.783036 678 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56478:
table { table_name: "default.Table_ccdc9aecb2d34778937160fc3d5cb4cc" } modify_external_catalogs: true
I20250628 02:04:50.783211 687 master_service.cc:948] ReplaceTablet: received request to replace tablet 48ccc4cdd28b422ca19674eb5285230d from {username='slave'} at 127.0.0.1:56478
I20250628 02:04:50.783282 688 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56478:
table { table_name: "default.Table_51acb52242224cffa3ce3ea99517351a" } modify_external_catalogs: true
I20250628 02:04:50.783389 688 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56478:
table { table_name: "default.Table_51acb52242224cffa3ce3ea99517351a" } modify_external_catalogs: true
I20250628 02:04:50.783601 683 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56478:
table { table_name: "default.Table_d40b0217227a48fc9431fb713c4ed6be" } modify_external_catalogs: true
I20250628 02:04:50.783624 682 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56478:
name: "default.Table_51147aa1de1d4fac95229bd0283919f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:50.783713 683 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56478:
table { table_name: "default.Table_d40b0217227a48fc9431fb713c4ed6be" } modify_external_catalogs: true
I20250628 02:04:50.783893 685 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56478:
table { table_name: "default.Table_25f7e6e53154401aa745623d0a9238e5" } new_table_name: "default.Table_2ffede105ab7483388016f792547fd7f" modify_external_catalogs: true
I20250628 02:04:50.783916 681 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56478:
name: "default.Table_f0ea02c8f8f944ebb5b8006e13d1f24c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:50.784121 680 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56478:
table { table_name: "default.Table_aeae2af4c3b941689431955a4b4cacbd" } modify_external_catalogs: true
I20250628 02:04:50.784215 680 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56478:
table { table_name: "default.Table_aeae2af4c3b941689431955a4b4cacbd" } modify_external_catalogs: true
I20250628 02:04:50.784399 679 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56478:
table { table_name: "default.Table_5f71bbbfb6c04933890499752142ceb3" } new_table_name: "default.Table_4d41a145f10944dca41a0675451382fc" modify_external_catalogs: true
I20250628 02:04:50.785928 886 raft_consensus.cc:1273] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 7 FOLLOWER]: Refusing update from remote peer 492448828d9a404fa245390eba0d56fb: Log matching property violated. Preceding OpId in replica: term: 5 index: 43. Preceding OpId from leader: term: 7 index: 49. (index mismatch)
I20250628 02:04:50.786514 959 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 43, Time since last communication: 0.000s
I20250628 02:04:50.790433 965 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:50.790572 951 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 492448828d9a404fa245390eba0d56fb. Latest consensus state: current_term: 7 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:50.791747 965 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:50.791781 951 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:50.792665 954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:50.792748 954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: LEADER
F20250628 02:04:50.792793 741 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 5 out of 10 received events: NotificationEvent(eventId=20, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_ccdc9aecb2d34778937160fc3d5cb4cc, message={"server":"","servicePrincipal":"","db":"default","table":"table_ccdc9aecb2d34778937160fc3d5cb4cc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ccdc9aecb2d34778937160fc3d5cb4cc\",\"kudu.table_id\":\"ae7629d68a7d4fdd9ed00f4b1b046cd3\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_0ba2136e171748f0ac70f94ad2378a15, message={"server":"","servicePrincipal":"","db":"default","table":"table_0ba2136e171748f0ac70f94ad2378a15","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0ba2136e171748f0ac70f94ad2378a15\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_0ba2136e171748f0ac70f94ad2378a15\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0ba2136e171748f0ac70f94ad2378a15\",\"kudu.table_id\":\"56f325f0d2044b03bda5136a7f717212\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1751076275, eventType=ALTER_TABLE, dbName=default, tableName=table_2ffede105ab7483388016f792547fd7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_25f7e6e53154401aa745623d0a9238e5","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_25f7e6e53154401aa745623d0a9238e5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_25f7e6e53154401aa745623d0a9238e5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_25f7e6e53154401aa745623d0a9238e5\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_2ffede105ab7483388016f792547fd7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_2ffede105ab7483388016f792547fd7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2ffede105ab7483388016f792547fd7f\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_48b5e32835b14bc489b2a0c3737240a3, message={"server":"","servicePrincipal":"","db":"default","table":"table_48b5e32835b14bc489b2a0c3737240a3","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_48b5e32835b14bc489b2a0c3737240a3\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_48b5e32835b14bc489b2a0c3737240a3\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_48b5e32835b14bc489b2a0c3737240a3\",\"kudu.table_id\":\"92619a7eecd54094bd8c3e908c2314e3\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_51acb52242224cffa3ce3ea99517351a, message={"server":"","servicePrincipal":"","db":"default","table":"table_51acb52242224cffa3ce3ea99517351a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_51acb52242224cffa3ce3ea99517351a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_51acb52242224cffa3ce3ea99517351a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_51acb52242224cffa3ce3ea99517351a\",\"kudu.table_id\":\"582401feb808444b885f2532faac36e0\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250628 02:04:50.793490 951 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:50.793571 951 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: FOLLOWER
W20250628 02:04:50.794525 671 consensus_peers.cc:489] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb -> Peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Couldn't send request to peer 36911adab67246cfbfaf77ef5221fc92. Status: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f7a933d6dcd google::LogMessage::Fail() at ??:0
I20250628 02:04:50.796535 963 mvcc.cc:204] Tried to move back new op lower bound from 7172408487054835712 to 7172408473736282112. Current Snapshot: MvccSnapshot[applied={T|T < 7172408474082971648}]
@ 0x7f7a933dab93 google::LogMessage::SendToLog() at ??:0
I20250628 02:04:50.797617 687 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Sending DeleteTablet for 3 replicas of tablet 48ccc4cdd28b422ca19674eb5285230d
I20250628 02:04:50.798746 687 catalog_manager.cc:6652] ReplaceTablet: tablet 48ccc4cdd28b422ca19674eb5285230d deleted and replaced by tablet 47667f26ca1e446dbd8e31d13410016c
I20250628 02:04:50.798970 687 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56478:
name: "default.Table_48b5e32835b14bc489b2a0c3737240a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f7a933d67cc google::LogMessage::Flush() at ??:0
@ 0x7f7a933d6a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f7a9836fc91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250628 02:04:50.810284 32367 tablet_service.cc:1515] Processing DeleteTablet for tablet 48ccc4cdd28b422ca19674eb5285230d with delete_type TABLET_DATA_DELETED (replaced by tablet 47667f26ca1e446dbd8e31d13410016c) from {username='slave'} at 127.0.0.1:36066
I20250628 02:04:50.810648 32369 tablet_service.cc:1468] Processing CreateTablet for tablet 47667f26ca1e446dbd8e31d13410016c (DEFAULT_TABLE table=default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f7a9836eba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250628 02:04:50.810884 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47667f26ca1e446dbd8e31d13410016c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:50.812302 32233 tablet_service.cc:1515] Processing DeleteTablet for tablet 48ccc4cdd28b422ca19674eb5285230d with delete_type TABLET_DATA_DELETED (replaced by tablet 47667f26ca1e446dbd8e31d13410016c) from {username='slave'} at 127.0.0.1:53602
I20250628 02:04:50.812633 32234 tablet_service.cc:1468] Processing CreateTablet for tablet 47667f26ca1e446dbd8e31d13410016c (DEFAULT_TABLE table=default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:50.812844 32234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47667f26ca1e446dbd8e31d13410016c. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f7a9836e30b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250628 02:04:50.813692 978 tablet_replica.cc:331] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:50.813796 978 raft_consensus.cc:2241] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [term 1 LEADER]: Raft consensus shutting down.
I20250628 02:04:50.813963 978 raft_consensus.cc:2270] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:50.814288 978 ts_tablet_manager.cc:1905] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:50.816388 978 ts_tablet_manager.cc:1918] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:50.816450 978 log.cc:1199] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/48ccc4cdd28b422ca19674eb5285230d
@ 0x7f7a98373a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250628 02:04:50.816723 978 ts_tablet_manager.cc:1939] T 48ccc4cdd28b422ca19674eb5285230d P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
I20250628 02:04:50.817409 977 tablet_replica.cc:331] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:50.817410 672 catalog_manager.cc:4928] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697): tablet 48ccc4cdd28b422ca19674eb5285230d (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]) successfully deleted
I20250628 02:04:50.817509 977 raft_consensus.cc:2241] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:50.817615 977 raft_consensus.cc:2270] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:50.817950 977 ts_tablet_manager.cc:1905] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:50.818552 979 tablet_bootstrap.cc:492] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:50.818585 980 tablet_bootstrap.cc:492] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:50.819375 980 tablet_bootstrap.cc:654] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:50.819751 979 tablet_bootstrap.cc:654] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:50.820783 980 tablet_bootstrap.cc:492] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:50.820845 980 ts_tablet_manager.cc:1397] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250628 02:04:50.821213 980 raft_consensus.cc:357] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:50.821302 980 raft_consensus.cc:383] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:50.821329 980 raft_consensus.cc:738] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:50.821455 980 consensus_queue.cc:260] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:50.821622 977 ts_tablet_manager.cc:1918] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:50.821674 977 log.cc:1199] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/48ccc4cdd28b422ca19674eb5285230d
I20250628 02:04:50.821719 32098 tablet_service.cc:1515] Processing DeleteTablet for tablet 48ccc4cdd28b422ca19674eb5285230d with delete_type TABLET_DATA_DELETED (replaced by tablet 47667f26ca1e446dbd8e31d13410016c) from {username='slave'} at 127.0.0.1:59582
I20250628 02:04:50.821909 977 ts_tablet_manager.cc:1939] T 48ccc4cdd28b422ca19674eb5285230d P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:04:50.822082 32099 tablet_service.cc:1468] Processing CreateTablet for tablet 47667f26ca1e446dbd8e31d13410016c (DEFAULT_TABLE table=default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:50.822325 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47667f26ca1e446dbd8e31d13410016c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:50.822430 670 catalog_manager.cc:4928] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733): tablet 48ccc4cdd28b422ca19674eb5285230d (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]) successfully deleted
I20250628 02:04:50.823120 980 ts_tablet_manager.cc:1428] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250628 02:04:50.823884 979 tablet_bootstrap.cc:492] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:50.823948 979 ts_tablet_manager.cc:1397] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250628 02:04:50.824332 979 raft_consensus.cc:357] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:50.824427 979 raft_consensus.cc:383] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:50.824455 979 raft_consensus.cc:738] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:50.824546 979 consensus_queue.cc:260] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:50.824785 979 ts_tablet_manager.cc:1428] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:50.827826 986 tablet_bootstrap.cc:492] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
@ 0x5638c7e0d752 std::function<>::operator()() at ??:0
I20250628 02:04:50.828558 986 tablet_bootstrap.cc:654] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:50.829207 985 tablet_replica.cc:331] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:50.829300 985 raft_consensus.cc:2241] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:50.829391 985 raft_consensus.cc:2270] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:50.829648 985 ts_tablet_manager.cc:1905] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f7a948280d1 kudu::Thread::SuperviseThread() at ??:0
I20250628 02:04:50.831254 985 ts_tablet_manager.cc:1918] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:50.831303 985 log.cc:1199] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/48ccc4cdd28b422ca19674eb5285230d
I20250628 02:04:50.831506 985 ts_tablet_manager.cc:1939] T 48ccc4cdd28b422ca19674eb5285230d P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:04:50.832463 669 catalog_manager.cc:4928] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393): tablet 48ccc4cdd28b422ca19674eb5285230d (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]) successfully deleted
@ 0x7f7a950176db start_thread at ??:0
I20250628 02:04:50.833863 986 tablet_bootstrap.cc:492] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:50.833925 986 ts_tablet_manager.cc:1397] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250628 02:04:50.834261 986 raft_consensus.cc:357] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:50.834334 986 raft_consensus.cc:383] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:50.834358 986 raft_consensus.cc:738] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:50.834460 986 consensus_queue.cc:260] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
@ 0x7f7a9258d71f clone at ??:0
I20250628 02:04:50.834671 986 ts_tablet_manager.cc:1428] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250628 02:04:50.958750 32053 connection.cc:537] client connection to 127.30.198.60:40321 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250628 02:04:50.960182 948 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:50.959791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.60:40321 (local address 127.30.198.3:34221)
0628 02:04:50.959899 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:50.959910 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0628 02:04:50.959991 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0628 02:04:50.960062 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42}
W20250628 02:04:50.960474 950 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:50.959362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.60:40321 (local address 127.30.198.1:49169)
0628 02:04:50.959521 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:50.959534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0628 02:04:50.959621 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0628 02:04:50.960363 (+ 742us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":99}
W20250628 02:04:50.960561 949 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:50.959362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.60:40321 (local address 127.30.198.2:55061)
0628 02:04:50.959521 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:50.959534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0628 02:04:50.959621 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0628 02:04:50.960467 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":82}
W20250628 02:04:50.960610 32443 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.30.198.60:40321)
W20250628 02:04:50.960860 32443 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.30.198.60:40321)
W20250628 02:04:50.960934 32443 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.198.60:40321)
W20250628 02:04:50.961126 32443 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.30.198.60:40321)
I20250628 02:04:51.176507 32539 raft_consensus.cc:491] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:51.176617 32539 raft_consensus.cc:513] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:51.177012 32539 leader_election.cc:290] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:51.177337 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47667f26ca1e446dbd8e31d13410016c" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:51.177342 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47667f26ca1e446dbd8e31d13410016c" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9" is_pre_election: true
I20250628 02:04:51.177467 32119 raft_consensus.cc:2466] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
I20250628 02:04:51.177484 32389 raft_consensus.cc:2466] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
I20250628 02:04:51.177739 32187 leader_election.cc:304] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:51.177886 32539 raft_consensus.cc:2802] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:51.177943 32539 raft_consensus.cc:491] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:51.177981 32539 raft_consensus.cc:3058] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:51.178821 32539 raft_consensus.cc:513] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:51.179124 32539 leader_election.cc:290] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Requested vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:51.179373 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47667f26ca1e446dbd8e31d13410016c" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9"
I20250628 02:04:51.179479 32119 raft_consensus.cc:3058] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:51.179476 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47667f26ca1e446dbd8e31d13410016c" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:51.179565 32389 raft_consensus.cc:3058] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:51.180377 32119 raft_consensus.cc:2466] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
I20250628 02:04:51.180377 32389 raft_consensus.cc:2466] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
I20250628 02:04:51.180630 32188 leader_election.cc:304] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:51.180766 32539 raft_consensus.cc:2802] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:51.180922 32539 raft_consensus.cc:695] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [term 1 LEADER]: Becoming Leader. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Running, Role: LEADER
I20250628 02:04:51.181048 32539 consensus_queue.cc:237] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:51.304930 999 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:04:51.305039 999 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:51.305368 999 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:51.306634 857 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:04:51.306715 858 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:04:51.306775 858 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:04:51.306891 999 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250628 02:04:51.624716 32539 consensus_queue.cc:1035] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:51.627882 32539 consensus_queue.cc:1035] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:52.025475 1011 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:04:52.025570 1011 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:52.025866 1011 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:52.027400 858 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:04:52.027819 857 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:52.027887 857 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:04:52.028021 1011 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250628 02:04:52.235409 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.61:35059
--webserver_interface=127.30.198.61
--webserver_port=39629
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250628 02:04:52.327854 1017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:04:52.328012 1017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:04:52.328047 1017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:04:52.330883 1017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 02:04:52.330936 1017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:04:52.330955 1017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 02:04:52.330971 1017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 02:04:52.334242 1017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.61:35059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.198.61
--webserver_port=39629
--never_fsync=true
--heap_profile_path=/tmp/kudu.1017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:04:52.335023 1017 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:04:52.335929 1017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 02:04:52.341444 1023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:52.341450 1022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:52.341594 1017 server_base.cc:1048] running on GCE node
W20250628 02:04:52.341531 1025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:52.342021 1017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:04:52.342753 1017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:04:52.343900 1017 hybrid_clock.cc:648] HybridClock initialized: now 1751076292343884 us; error 33 us; skew 500 ppm
I20250628 02:04:52.349931 1017 webserver.cc:469] Webserver started at http://127.30.198.61:39629/ using document root <none> and password file <none>
I20250628 02:04:52.350414 1017 fs_manager.cc:362] Metadata directory not provided
I20250628 02:04:52.350463 1017 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:04:52.353251 1017 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250628 02:04:52.354732 1031 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:52.355494 1017 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250628 02:04:52.355584 1017 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
uuid: "36911adab67246cfbfaf77ef5221fc92"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:52.355914 1017 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:04:52.384012 1017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:04:52.384479 1017 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:04:52.384614 1017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:04:52.390528 1017 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.61:35059
I20250628 02:04:52.390550 1083 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.61:35059 every 8 connection(s)
I20250628 02:04:52.391443 1017 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-1/data/info.pb
I20250628 02:04:52.395458 1084 sys_catalog.cc:263] Verifying existing consensus state
I20250628 02:04:52.396895 1084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Bootstrap starting.
I20250628 02:04:52.400556 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 1017
I20250628 02:04:52.404731 1084 log.cc:826] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Log is configured to *not* fsync() on all Append() calls
I20250628 02:04:52.425942 1084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Bootstrap replayed 1/1 log segments. Stats: ops{read=47 overwritten=0 applied=47 ignored=0} inserts{seen=30 ignored=0} mutations{seen=31 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 02:04:52.426224 1084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Bootstrap complete.
I20250628 02:04:52.428393 1084 raft_consensus.cc:357] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:52.428658 1084 raft_consensus.cc:738] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36911adab67246cfbfaf77ef5221fc92, State: Initialized, Role: FOLLOWER
I20250628 02:04:52.429005 1084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 47, Last appended: 6.47, Last appended by leader: 47, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:52.429329 1089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:52.429435 1089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:52.429839 1084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 02:04:52.434643 1104 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:52.434741 1104 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: loading cluster ID for follower catalog manager: success
I20250628 02:04:52.436981 1104 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: acquiring CA information for follower catalog manager: success
I20250628 02:04:52.438488 1104 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250628 02:04:52.444012 1017 master_runner.cc:186] Error getting master registration for 127.30.198.60:40321: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111),
I20250628 02:04:52.445945 1017 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:04:52.531351 32302 heartbeater.cc:344] Connected to a master server at 127.30.198.61:35059
I20250628 02:04:52.532985 1048 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" instance_seqno: 1751076273975274) as {username='slave'} at 127.30.198.2:46515; Asking this server to re-register.
I20250628 02:04:52.533532 32302 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:52.534566 1048 ts_manager.cc:194] Registered new tserver with Master: b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:52.633710 32437 heartbeater.cc:344] Connected to a master server at 127.30.198.61:35059
I20250628 02:04:52.634639 1048 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" instance_seqno: 1751076274129070) as {username='slave'} at 127.30.198.3:41921; Asking this server to re-register.
I20250628 02:04:52.634918 32167 heartbeater.cc:344] Connected to a master server at 127.30.198.61:35059
I20250628 02:04:52.634972 32437 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:52.635661 1047 ts_manager.cc:194] Registered new tserver with Master: d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:52.636059 1048 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cb3d5d42c47546258deed617114e71f9" instance_seqno: 1751076273823804) as {username='slave'} at 127.30.198.1:60659; Asking this server to re-register.
I20250628 02:04:52.636445 32167 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:52.637200 1048 ts_manager.cc:194] Registered new tserver with Master: cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:52.871330 1089 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:52.871528 1089 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:52.872777 1089 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
I20250628 02:04:52.873140 886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36911adab67246cfbfaf77ef5221fc92" candidate_term: 7 candidate_status { last_received { term: 6 index: 47 } } ignore_live_leader: false dest_uuid: "e9fd188e41da4ec0b493c144accb0344" is_pre_election: true
W20250628 02:04:52.873219 1033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:52.873332 886 raft_consensus.cc:2391] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 36911adab67246cfbfaf77ef5221fc92 in current term 7: Already voted for candidate 492448828d9a404fa245390eba0d56fb in this term.
W20250628 02:04:52.874034 1033 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:52.874109 1033 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:52.874329 1089 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 6 FOLLOWER]: Advancing to term 7
I20250628 02:04:52.876169 1089 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250628 02:04:53.109275 1116 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:04:53.109382 1116 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:53.109696 1116 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:53.110746 857 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:53.112946 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 8 candidate_status { last_received { term: 7 index: 50 } } ignore_live_leader: false dest_uuid: "36911adab67246cfbfaf77ef5221fc92" is_pre_election: true
I20250628 02:04:53.113170 1058 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fd188e41da4ec0b493c144accb0344 in term 7.
I20250628 02:04:53.113445 858 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344; no voters: 492448828d9a404fa245390eba0d56fb
I20250628 02:04:53.113602 1116 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250628 02:04:53.113656 1116 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:04:53.113684 1116 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 7 FOLLOWER]: Advancing to term 8
I20250628 02:04:53.114416 1116 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:53.114709 1116 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 8 election: Requested vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
I20250628 02:04:53.114980 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 8 candidate_status { last_received { term: 7 index: 50 } } ignore_live_leader: false dest_uuid: "36911adab67246cfbfaf77ef5221fc92"
I20250628 02:04:53.115087 1058 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 7 FOLLOWER]: Advancing to term 8
W20250628 02:04:53.115561 857 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:53.116077 1058 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fd188e41da4ec0b493c144accb0344 in term 8.
I20250628 02:04:53.116315 858 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344; no voters: 492448828d9a404fa245390eba0d56fb
I20250628 02:04:53.116477 1116 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 8 FOLLOWER]: Leader election won for term 8
I20250628 02:04:53.116653 1116 raft_consensus.cc:695] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 8 LEADER]: Becoming Leader. State: Replica: e9fd188e41da4ec0b493c144accb0344, State: Running, Role: LEADER
I20250628 02:04:53.116930 1116 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 7.50, Last appended by leader: 50, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:53.117846 1120 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fd188e41da4ec0b493c144accb0344. Latest consensus state: current_term: 8 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:53.117938 1120 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:53.118183 1122 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 02:04:53.119551 1122 catalog_manager.cc:671] Loaded metadata for table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]
I20250628 02:04:53.119836 1122 catalog_manager.cc:671] Loaded metadata for table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]
I20250628 02:04:53.120033 1122 catalog_manager.cc:671] Loaded metadata for table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]
I20250628 02:04:53.120239 1122 catalog_manager.cc:671] Loaded metadata for table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]
I20250628 02:04:53.120429 1122 catalog_manager.cc:671] Loaded metadata for table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]
I20250628 02:04:53.120605 1122 catalog_manager.cc:671] Loaded metadata for table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]
I20250628 02:04:53.120790 1122 catalog_manager.cc:671] Loaded metadata for table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]
I20250628 02:04:53.121851 1122 tablet_loader.cc:96] loaded metadata for tablet 47667f26ca1e446dbd8e31d13410016c (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3])
I20250628 02:04:53.122228 1122 tablet_loader.cc:96] loaded metadata for tablet 73d2f50340b14bd08abd193108c40b47 (table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc])
I20250628 02:04:53.122372 1122 tablet_loader.cc:96] loaded metadata for tablet 8c79c884094b453b90c79f0873169071 (table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c])
I20250628 02:04:53.122517 1122 tablet_loader.cc:96] loaded metadata for tablet ac08335565ca46638e022c4588a592f0 (table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0])
I20250628 02:04:53.122855 1122 tablet_loader.cc:96] loaded metadata for tablet bc437af4ee094529ae0078641d819bbc (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e])
I20250628 02:04:53.122992 1122 tablet_loader.cc:96] loaded metadata for tablet d857afe694cf4efb9efc415fe5476691 (table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003])
I20250628 02:04:53.123210 1122 tablet_loader.cc:96] loaded metadata for tablet e6af82fc366e4121967e834148998420 (table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb])
I20250628 02:04:53.123309 1122 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 02:04:53.123728 1122 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:53.123770 1122 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 02:04:53.124281 1122 catalog_manager.cc:1506] Loading token signing keys...
I20250628 02:04:53.124678 1122 catalog_manager.cc:5966] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Loaded TSK: 0
I20250628 02:04:53.125281 1122 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 02:04:53.125684 1122 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250628 02:04:53.126071 1122 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 19
I20250628 02:04:53.174360 876 master_service.cc:948] ReplaceTablet: received request to replace tablet 8c79c884094b453b90c79f0873169071 from {username='slave'} at 127.0.0.1:52446
I20250628 02:04:53.174403 875 master_service.cc:948] ReplaceTablet: received request to replace tablet d857afe694cf4efb9efc415fe5476691 from {username='slave'} at 127.0.0.1:52446
I20250628 02:04:53.174609 874 master_service.cc:948] ReplaceTablet: received request to replace tablet ac08335565ca46638e022c4588a592f0 from {username='slave'} at 127.0.0.1:52446
I20250628 02:04:53.174841 873 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52446:
table { table_name: "default.Table_48852943c5c64ddfbde3e614d034c8d6" } modify_external_catalogs: true
I20250628 02:04:53.174959 873 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52446:
table { table_name: "default.Table_48852943c5c64ddfbde3e614d034c8d6" } modify_external_catalogs: true
I20250628 02:04:53.175319 869 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52446:
name: "default.Table_51147aa1de1d4fac95229bd0283919f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:53.175516 871 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52446:
name: "default.Table_4687199de5174487b23a117d7fe19211"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:53.175340 872 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52446:
name: "default.Table_7eb9297b9cef4c8e82bbdb793ba152cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:53.175638 867 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52446:
name: "default.Table_0ba2136e171748f0ac70f94ad2378a15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:53.175902 870 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52446:
table { table_name: "default.Table_d40b0217227a48fc9431fb713c4ed6be" } modify_external_catalogs: true
I20250628 02:04:53.174961 868 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52446:
table { table_name: "default.Table_25f7e6e53154401aa745623d0a9238e5" } new_table_name: "default.Table_2ffede105ab7483388016f792547fd7f" modify_external_catalogs: true
I20250628 02:04:53.175987 870 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52446:
table { table_name: "default.Table_d40b0217227a48fc9431fb713c4ed6be" } modify_external_catalogs: true
W20250628 02:04:53.178829 857 consensus_peers.cc:489] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 -> Peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Couldn't send request to peer 492448828d9a404fa245390eba0d56fb. Status: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250628 02:04:53.178828 1058 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 8 FOLLOWER]: Refusing update from remote peer e9fd188e41da4ec0b493c144accb0344: Log matching property violated. Preceding OpId in replica: term: 6 index: 47. Preceding OpId from leader: term: 8 index: 54. (index mismatch)
I20250628 02:04:53.179317 1120 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 47, Time since last communication: 0.000s
F20250628 02:04:53.181423 929 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 5 out of 10 received events: NotificationEvent(eventId=20, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_ccdc9aecb2d34778937160fc3d5cb4cc, message={"server":"","servicePrincipal":"","db":"default","table":"table_ccdc9aecb2d34778937160fc3d5cb4cc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ccdc9aecb2d34778937160fc3d5cb4cc\",\"kudu.table_id\":\"ae7629d68a7d4fdd9ed00f4b1b046cd3\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_0ba2136e171748f0ac70f94ad2378a15, message={"server":"","servicePrincipal":"","db":"default","table":"table_0ba2136e171748f0ac70f94ad2378a15","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0ba2136e171748f0ac70f94ad2378a15\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_0ba2136e171748f0ac70f94ad2378a15\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0ba2136e171748f0ac70f94ad2378a15\",\"kudu.table_id\":\"56f325f0d2044b03bda5136a7f717212\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1751076275, eventType=ALTER_TABLE, dbName=default, tableName=table_2ffede105ab7483388016f792547fd7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_25f7e6e53154401aa745623d0a9238e5","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_25f7e6e53154401aa745623d0a9238e5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_25f7e6e53154401aa745623d0a9238e5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_25f7e6e53154401aa745623d0a9238e5\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_2ffede105ab7483388016f792547fd7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_2ffede105ab7483388016f792547fd7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2ffede105ab7483388016f792547fd7f\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_48b5e32835b14bc489b2a0c3737240a3, message={"server":"","servicePrincipal":"","db":"default","table":"table_48b5e32835b14bc489b2a0c3737240a3","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_48b5e32835b14bc489b2a0c3737240a3\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_48b5e32835b14bc489b2a0c3737240a3\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_48b5e32835b14bc489b2a0c3737240a3\",\"kudu.table_id\":\"92619a7eecd54094bd8c3e908c2314e3\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_51acb52242224cffa3ce3ea99517351a, message={"server":"","servicePrincipal":"","db":"default","table":"table_51acb52242224cffa3ce3ea99517351a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_51acb52242224cffa3ce3ea99517351a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_51acb52242224cffa3ce3ea99517351a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_51acb52242224cffa3ce3ea99517351a\",\"kudu.table_id\":\"582401feb808444b885f2532faac36e0\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250628 02:04:53.183655 1089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fd188e41da4ec0b493c144accb0344. Latest consensus state: current_term: 8 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:53.183802 1089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f2ee39bbdcd google::LogMessage::Fail() at ??:0
I20250628 02:04:53.184672 1126 mvcc.cc:204] Tried to move back new op lower bound from 7172408487121756160 to 7172408486843924480. Current Snapshot: MvccSnapshot[applied={T|T < 7172408487121756160}]
I20250628 02:04:53.184798 1128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:53.184897 1128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:53.186136 1120 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:53.186240 1120 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: LEADER
@ 0x7f2ee39bfb93 google::LogMessage::SendToLog() at ??:0
I20250628 02:04:53.186939 876 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Sending DeleteTablet for 3 replicas of tablet 8c79c884094b453b90c79f0873169071
I20250628 02:04:53.187095 874 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Sending DeleteTablet for 3 replicas of tablet ac08335565ca46638e022c4588a592f0
I20250628 02:04:53.188024 874 catalog_manager.cc:6652] ReplaceTablet: tablet ac08335565ca46638e022c4588a592f0 deleted and replaced by tablet 7d751e2d40b94c3ebf4faaa9e621fc20
I20250628 02:04:53.188271 874 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52446:
name: "default.Table_f0ea02c8f8f944ebb5b8006e13d1f24c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:53.188716 876 catalog_manager.cc:6652] ReplaceTablet: tablet 8c79c884094b453b90c79f0873169071 deleted and replaced by tablet 31216cf12a464b769be633b41088c417
I20250628 02:04:53.188745 1128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:53.188838 1128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:53.188875 876 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52446:
table { table_name: "default.Table_aeae2af4c3b941689431955a4b4cacbd" } modify_external_catalogs: true
I20250628 02:04:53.188958 876 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52446:
table { table_name: "default.Table_aeae2af4c3b941689431955a4b4cacbd" } modify_external_catalogs: true
I20250628 02:04:53.189225 875 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Sending DeleteTablet for 3 replicas of tablet d857afe694cf4efb9efc415fe5476691
I20250628 02:04:53.189482 875 catalog_manager.cc:6652] ReplaceTablet: tablet d857afe694cf4efb9efc415fe5476691 deleted and replaced by tablet 70e6b67d5b87462a96838d8229f0e13a
I20250628 02:04:53.189640 875 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52446:
table { table_name: "default.Table_5f71bbbfb6c04933890499752142ceb3" } new_table_name: "default.Table_4d41a145f10944dca41a0675451382fc" modify_external_catalogs: true
@ 0x7f2ee39bb7cc google::LogMessage::Flush() at ??:0
@ 0x7f2ee39bba11 google::LogMessage::~LogMessage() at ??:0
I20250628 02:04:53.194927 32099 tablet_service.cc:1515] Processing DeleteTablet for tablet 8c79c884094b453b90c79f0873169071 with delete_type TABLET_DATA_DELETED (replaced by tablet 31216cf12a464b769be633b41088c417) from {username='slave'} at 127.0.0.1:59586
I20250628 02:04:53.195024 32098 tablet_service.cc:1515] Processing DeleteTablet for tablet ac08335565ca46638e022c4588a592f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d751e2d40b94c3ebf4faaa9e621fc20) from {username='slave'} at 127.0.0.1:59586
I20250628 02:04:53.195209 32097 tablet_service.cc:1515] Processing DeleteTablet for tablet d857afe694cf4efb9efc415fe5476691 with delete_type TABLET_DATA_DELETED (replaced by tablet 70e6b67d5b87462a96838d8229f0e13a) from {username='slave'} at 127.0.0.1:59586
@ 0x7f2ee8954c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250628 02:04:53.196713 32097 tablet_service.cc:1468] Processing CreateTablet for tablet 7d751e2d40b94c3ebf4faaa9e621fc20 (DEFAULT_TABLE table=default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:53.197036 32097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d751e2d40b94c3ebf4faaa9e621fc20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:53.199714 1140 tablet_replica.cc:331] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:53.199843 1140 raft_consensus.cc:2241] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Raft consensus shutting down.
I20250628 02:04:53.200033 1140 raft_consensus.cc:2270] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f2ee8953ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250628 02:04:53.200439 1140 ts_tablet_manager.cc:1905] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:53.201304 32369 tablet_service.cc:1515] Processing DeleteTablet for tablet ac08335565ca46638e022c4588a592f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d751e2d40b94c3ebf4faaa9e621fc20) from {username='slave'} at 127.0.0.1:36082
I20250628 02:04:53.201541 32367 tablet_service.cc:1515] Processing DeleteTablet for tablet 8c79c884094b453b90c79f0873169071 with delete_type TABLET_DATA_DELETED (replaced by tablet 31216cf12a464b769be633b41088c417) from {username='slave'} at 127.0.0.1:36082
I20250628 02:04:53.201650 32368 tablet_service.cc:1515] Processing DeleteTablet for tablet d857afe694cf4efb9efc415fe5476691 with delete_type TABLET_DATA_DELETED (replaced by tablet 70e6b67d5b87462a96838d8229f0e13a) from {username='slave'} at 127.0.0.1:36082
I20250628 02:04:53.201900 32099 tablet_service.cc:1468] Processing CreateTablet for tablet 31216cf12a464b769be633b41088c417 (DEFAULT_TABLE table=default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:53.202029 32366 tablet_service.cc:1468] Processing CreateTablet for tablet 7d751e2d40b94c3ebf4faaa9e621fc20 (DEFAULT_TABLE table=default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:53.202101 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31216cf12a464b769be633b41088c417. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:53.202253 32366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d751e2d40b94c3ebf4faaa9e621fc20. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f2ee895330b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250628 02:04:53.204557 32365 tablet_service.cc:1468] Processing CreateTablet for tablet 31216cf12a464b769be633b41088c417 (DEFAULT_TABLE table=default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:53.204783 32365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31216cf12a464b769be633b41088c417. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:53.205070 32099 tablet_service.cc:1468] Processing CreateTablet for tablet 70e6b67d5b87462a96838d8229f0e13a (DEFAULT_TABLE table=default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:53.205278 32099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e6b67d5b87462a96838d8229f0e13a. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f2ee8958a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x558d469e1752 std::function<>::operator()() at ??:0
I20250628 02:04:53.207546 32365 tablet_service.cc:1468] Processing CreateTablet for tablet 70e6b67d5b87462a96838d8229f0e13a (DEFAULT_TABLE table=default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:53.207789 32365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e6b67d5b87462a96838d8229f0e13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:53.208381 32231 tablet_service.cc:1515] Processing DeleteTablet for tablet d857afe694cf4efb9efc415fe5476691 with delete_type TABLET_DATA_DELETED (replaced by tablet 70e6b67d5b87462a96838d8229f0e13a) from {username='slave'} at 127.0.0.1:53612
I20250628 02:04:53.208442 32234 tablet_service.cc:1515] Processing DeleteTablet for tablet ac08335565ca46638e022c4588a592f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d751e2d40b94c3ebf4faaa9e621fc20) from {username='slave'} at 127.0.0.1:53612
I20250628 02:04:53.208500 32230 tablet_service.cc:1468] Processing CreateTablet for tablet 31216cf12a464b769be633b41088c417 (DEFAULT_TABLE table=default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:53.208619 32233 tablet_service.cc:1515] Processing DeleteTablet for tablet 8c79c884094b453b90c79f0873169071 with delete_type TABLET_DATA_DELETED (replaced by tablet 31216cf12a464b769be633b41088c417) from {username='slave'} at 127.0.0.1:53612
I20250628 02:04:53.208741 32230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31216cf12a464b769be633b41088c417. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:53.208897 32232 tablet_service.cc:1468] Processing CreateTablet for tablet 7d751e2d40b94c3ebf4faaa9e621fc20 (DEFAULT_TABLE table=default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:53.209156 32232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d751e2d40b94c3ebf4faaa9e621fc20. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f2ee4e0d0d1 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2ee55fc6db start_thread at ??:0
I20250628 02:04:53.211334 32229 tablet_service.cc:1468] Processing CreateTablet for tablet 70e6b67d5b87462a96838d8229f0e13a (DEFAULT_TABLE table=default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:53.211592 32229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e6b67d5b87462a96838d8229f0e13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:53.212172 1140 ts_tablet_manager.cc:1918] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:53.212242 1140 log.cc:1199] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/8c79c884094b453b90c79f0873169071
I20250628 02:04:53.212538 1140 ts_tablet_manager.cc:1939] T 8c79c884094b453b90c79f0873169071 P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:04:53.212831 1142 tablet_replica.cc:331] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:53.212944 1142 raft_consensus.cc:2241] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:53.213033 1142 raft_consensus.cc:2270] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f2ee2b7271f clone at ??:0
I20250628 02:04:53.213243 1140 tablet_replica.cc:331] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:53.213646 1140 raft_consensus.cc:2241] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Raft consensus shutting down.
I20250628 02:04:53.213810 1140 raft_consensus.cc:2270] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:53.213937 1142 ts_tablet_manager.cc:1905] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:53.213966 1146 tablet_bootstrap.cc:492] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:53.214136 1140 ts_tablet_manager.cc:1905] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:53.214625 1144 tablet_bootstrap.cc:492] T 7d751e2d40b94c3ebf4faaa9e621fc20 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:53.214977 1146 tablet_bootstrap.cc:654] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:53.215627 1145 tablet_replica.cc:331] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:53.215760 1145 raft_consensus.cc:2241] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:53.215857 1145 raft_consensus.cc:2270] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:53.216099 1142 ts_tablet_manager.cc:1918] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:53.216181 1142 log.cc:1199] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/ac08335565ca46638e022c4588a592f0
I20250628 02:04:53.216185 1145 ts_tablet_manager.cc:1905] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:53.216487 1142 ts_tablet_manager.cc:1939] T ac08335565ca46638e022c4588a592f0 P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:04:53.217028 1142 tablet_replica.cc:331] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:53.217149 1142 raft_consensus.cc:2241] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:53.217245 1142 raft_consensus.cc:2270] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:53.217363 1140 ts_tablet_manager.cc:1918] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:53.217440 1140 log.cc:1199] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/ac08335565ca46638e022c4588a592f0
I20250628 02:04:53.217623 1142 ts_tablet_manager.cc:1905] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:53.217690 1140 ts_tablet_manager.cc:1939] T ac08335565ca46638e022c4588a592f0 P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:04:53.218400 1144 tablet_bootstrap.cc:654] T 7d751e2d40b94c3ebf4faaa9e621fc20 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:53.218634 1145 ts_tablet_manager.cc:1918] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:53.218695 1145 log.cc:1199] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/d857afe694cf4efb9efc415fe5476691
I20250628 02:04:53.218963 1145 ts_tablet_manager.cc:1939] T d857afe694cf4efb9efc415fe5476691 P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
I20250628 02:04:53.219015 1140 tablet_replica.cc:331] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:53.219126 1140 raft_consensus.cc:2241] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:53.219223 1140 raft_consensus.cc:2270] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:53.219357 1146 tablet_bootstrap.cc:492] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:53.219470 1146 ts_tablet_manager.cc:1397] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250628 02:04:53.219519 1145 tablet_replica.cc:331] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:53.219542 1140 ts_tablet_manager.cc:1905] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:53.219614 1145 raft_consensus.cc:2241] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:53.219722 1143 tablet_bootstrap.cc:492] T 7d751e2d40b94c3ebf4faaa9e621fc20 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:53.219722 1145 raft_consensus.cc:2270] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:53.219954 1146 raft_consensus.cc:357] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.220041 1145 ts_tablet_manager.cc:1905] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:53.220041 1146 raft_consensus.cc:383] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:53.220572 1146 raft_consensus.cc:738] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:53.220723 1146 consensus_queue.cc:260] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.220990 1143 tablet_bootstrap.cc:654] T 7d751e2d40b94c3ebf4faaa9e621fc20 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:53.221276 1140 ts_tablet_manager.cc:1918] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:53.221333 1140 log.cc:1199] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/d857afe694cf4efb9efc415fe5476691
I20250628 02:04:53.221556 1140 ts_tablet_manager.cc:1939] T d857afe694cf4efb9efc415fe5476691 P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:04:53.221606 1142 ts_tablet_manager.cc:1918] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:53.221664 1142 log.cc:1199] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/8c79c884094b453b90c79f0873169071
I20250628 02:04:53.221890 1142 ts_tablet_manager.cc:1939] T 8c79c884094b453b90c79f0873169071 P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:04:53.221915 1145 ts_tablet_manager.cc:1918] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:53.221971 1145 log.cc:1199] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/ac08335565ca46638e022c4588a592f0
I20250628 02:04:53.222208 1145 ts_tablet_manager.cc:1939] T ac08335565ca46638e022c4588a592f0 P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
I20250628 02:04:53.222364 1142 tablet_replica.cc:331] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:53.222456 1142 raft_consensus.cc:2241] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [term 1 LEADER]: Raft consensus shutting down.
I20250628 02:04:53.222651 1145 tablet_replica.cc:331] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:53.222733 1145 raft_consensus.cc:2241] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:53.222826 1145 raft_consensus.cc:2270] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:53.223179 1145 ts_tablet_manager.cc:1905] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:53.223450 1146 ts_tablet_manager.cc:1428] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250628 02:04:53.223773 1142 raft_consensus.cc:2270] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:53.224107 1142 ts_tablet_manager.cc:1905] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:53.224905 1145 ts_tablet_manager.cc:1918] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:53.224967 1145 log.cc:1199] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/8c79c884094b453b90c79f0873169071
I20250628 02:04:53.225215 1145 ts_tablet_manager.cc:1939] T 8c79c884094b453b90c79f0873169071 P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
I20250628 02:04:53.225327 1146 tablet_bootstrap.cc:492] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:53.225893 1142 ts_tablet_manager.cc:1918] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:53.225962 1142 log.cc:1199] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/d857afe694cf4efb9efc415fe5476691
I20250628 02:04:53.226215 1142 ts_tablet_manager.cc:1939] T d857afe694cf4efb9efc415fe5476691 P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:04:53.226303 1146 tablet_bootstrap.cc:654] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:53.227392 1146 tablet_bootstrap.cc:492] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:53.227479 1146 ts_tablet_manager.cc:1397] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:53.227532 1144 tablet_bootstrap.cc:492] T 7d751e2d40b94c3ebf4faaa9e621fc20 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:53.227617 1144 ts_tablet_manager.cc:1397] T 7d751e2d40b94c3ebf4faaa9e621fc20 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250628 02:04:53.228047 1146 raft_consensus.cc:357] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:53.228101 1144 raft_consensus.cc:357] T 7d751e2d40b94c3ebf4faaa9e621fc20 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:53.228152 1146 raft_consensus.cc:383] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:53.228195 1144 raft_consensus.cc:383] T 7d751e2d40b94c3ebf4faaa9e621fc20 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:53.228193 1146 raft_consensus.cc:738] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:53.228228 1144 raft_consensus.cc:738] T 7d751e2d40b94c3ebf4faaa9e621fc20 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:53.228315 1146 consensus_queue.cc:260] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:53.228338 1144 consensus_queue.cc:260] T 7d751e2d40b94c3ebf4faaa9e621fc20 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:53.228585 1144 ts_tablet_manager.cc:1428] T 7d751e2d40b94c3ebf4faaa9e621fc20 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250628 02:04:53.228641 1146 ts_tablet_manager.cc:1428] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:53.228780 1144 tablet_bootstrap.cc:492] T 31216cf12a464b769be633b41088c417 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:53.228833 1146 tablet_bootstrap.cc:492] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:53.229689 1146 tablet_bootstrap.cc:654] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:53.229916 32436 heartbeater.cc:499] Master 127.30.198.62:42671 was elected leader, sending a full tablet report...
I20250628 02:04:53.230325 1144 tablet_bootstrap.cc:654] T 31216cf12a464b769be633b41088c417 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:53.232396 1143 tablet_bootstrap.cc:492] T 7d751e2d40b94c3ebf4faaa9e621fc20 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:53.232467 1143 ts_tablet_manager.cc:1397] T 7d751e2d40b94c3ebf4faaa9e621fc20 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250628 02:04:53.232939 1143 raft_consensus.cc:357] T 7d751e2d40b94c3ebf4faaa9e621fc20 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:53.233032 1143 raft_consensus.cc:383] T 7d751e2d40b94c3ebf4faaa9e621fc20 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:53.233062 1143 raft_consensus.cc:738] T 7d751e2d40b94c3ebf4faaa9e621fc20 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:53.233189 1143 consensus_queue.cc:260] T 7d751e2d40b94c3ebf4faaa9e621fc20 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:53.233428 1143 ts_tablet_manager.cc:1428] T 7d751e2d40b94c3ebf4faaa9e621fc20 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:53.233526 32166 heartbeater.cc:499] Master 127.30.198.62:42671 was elected leader, sending a full tablet report...
I20250628 02:04:53.233588 1143 tablet_bootstrap.cc:492] T 31216cf12a464b769be633b41088c417 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:53.234217 1146 tablet_bootstrap.cc:492] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:53.234278 1146 ts_tablet_manager.cc:1397] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20250628 02:04:53.234546 1143 tablet_bootstrap.cc:654] T 31216cf12a464b769be633b41088c417 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:53.235273 1146 raft_consensus.cc:357] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.235357 1146 raft_consensus.cc:383] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:53.235395 1146 raft_consensus.cc:738] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:53.235495 1146 consensus_queue.cc:260] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.237120 1146 ts_tablet_manager.cc:1428] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250628 02:04:53.237898 1144 tablet_bootstrap.cc:492] T 31216cf12a464b769be633b41088c417 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:53.237958 1144 ts_tablet_manager.cc:1397] T 31216cf12a464b769be633b41088c417 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20250628 02:04:53.238349 1144 raft_consensus.cc:357] T 31216cf12a464b769be633b41088c417 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.238438 1144 raft_consensus.cc:383] T 31216cf12a464b769be633b41088c417 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:53.238466 1144 raft_consensus.cc:738] T 31216cf12a464b769be633b41088c417 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:53.238561 1144 consensus_queue.cc:260] T 31216cf12a464b769be633b41088c417 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.238788 1144 ts_tablet_manager.cc:1428] T 31216cf12a464b769be633b41088c417 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250628 02:04:53.238993 1144 tablet_bootstrap.cc:492] T 70e6b67d5b87462a96838d8229f0e13a P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:53.239941 1144 tablet_bootstrap.cc:654] T 70e6b67d5b87462a96838d8229f0e13a P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:53.240845 1144 tablet_bootstrap.cc:492] T 70e6b67d5b87462a96838d8229f0e13a P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:53.240919 1144 ts_tablet_manager.cc:1397] T 70e6b67d5b87462a96838d8229f0e13a P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250628 02:04:53.241394 1144 raft_consensus.cc:357] T 70e6b67d5b87462a96838d8229f0e13a P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.241497 1144 raft_consensus.cc:383] T 70e6b67d5b87462a96838d8229f0e13a P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:53.241523 1144 raft_consensus.cc:738] T 70e6b67d5b87462a96838d8229f0e13a P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:53.241642 1144 consensus_queue.cc:260] T 70e6b67d5b87462a96838d8229f0e13a P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.241947 1144 ts_tablet_manager.cc:1428] T 70e6b67d5b87462a96838d8229f0e13a P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250628 02:04:53.242705 1143 tablet_bootstrap.cc:492] T 31216cf12a464b769be633b41088c417 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:53.242777 1143 ts_tablet_manager.cc:1397] T 31216cf12a464b769be633b41088c417 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20250628 02:04:53.243322 1143 raft_consensus.cc:357] T 31216cf12a464b769be633b41088c417 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.243414 1143 raft_consensus.cc:383] T 31216cf12a464b769be633b41088c417 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:53.243444 1143 raft_consensus.cc:738] T 31216cf12a464b769be633b41088c417 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:53.243588 1143 consensus_queue.cc:260] T 31216cf12a464b769be633b41088c417 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.244966 1143 ts_tablet_manager.cc:1428] T 31216cf12a464b769be633b41088c417 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250628 02:04:53.245769 1143 tablet_bootstrap.cc:492] T 70e6b67d5b87462a96838d8229f0e13a P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:53.246631 1143 tablet_bootstrap.cc:654] T 70e6b67d5b87462a96838d8229f0e13a P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:53.247859 1143 tablet_bootstrap.cc:492] T 70e6b67d5b87462a96838d8229f0e13a P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:53.247951 1143 ts_tablet_manager.cc:1397] T 70e6b67d5b87462a96838d8229f0e13a P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250628 02:04:53.248344 1143 raft_consensus.cc:357] T 70e6b67d5b87462a96838d8229f0e13a P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.248430 1143 raft_consensus.cc:383] T 70e6b67d5b87462a96838d8229f0e13a P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:53.248461 1143 raft_consensus.cc:738] T 70e6b67d5b87462a96838d8229f0e13a P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:53.248579 1143 consensus_queue.cc:260] T 70e6b67d5b87462a96838d8229f0e13a P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.248930 1143 ts_tablet_manager.cc:1428] T 70e6b67d5b87462a96838d8229f0e13a P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:53.286676 1006 raft_consensus.cc:491] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:53.286801 1006 raft_consensus.cc:513] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.287266 1006 leader_election.cc:290] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733), cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:53.287607 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70e6b67d5b87462a96838d8229f0e13a" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:53.287614 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70e6b67d5b87462a96838d8229f0e13a" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9" is_pre_election: true
I20250628 02:04:53.287768 32389 raft_consensus.cc:2466] T 70e6b67d5b87462a96838d8229f0e13a P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
I20250628 02:04:53.287801 32119 raft_consensus.cc:2466] T 70e6b67d5b87462a96838d8229f0e13a P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
I20250628 02:04:53.288054 32188 leader_election.cc:304] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:53.288223 1006 raft_consensus.cc:2802] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:53.288278 1006 raft_consensus.cc:491] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:53.288318 1006 raft_consensus.cc:3058] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:53.289013 1006 raft_consensus.cc:513] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.289269 1006 leader_election.cc:290] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Requested vote from peers d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733), cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:53.289561 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70e6b67d5b87462a96838d8229f0e13a" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9"
I20250628 02:04:53.289569 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70e6b67d5b87462a96838d8229f0e13a" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:53.289659 32389 raft_consensus.cc:3058] T 70e6b67d5b87462a96838d8229f0e13a P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:53.289659 32119 raft_consensus.cc:3058] T 70e6b67d5b87462a96838d8229f0e13a P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:53.290463 32389 raft_consensus.cc:2466] T 70e6b67d5b87462a96838d8229f0e13a P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
I20250628 02:04:53.290463 32119 raft_consensus.cc:2466] T 70e6b67d5b87462a96838d8229f0e13a P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
I20250628 02:04:53.290722 32187 leader_election.cc:304] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:53.290877 1006 raft_consensus.cc:2802] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:53.291042 1006 raft_consensus.cc:695] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [term 1 LEADER]: Becoming Leader. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Running, Role: LEADER
I20250628 02:04:53.291173 1006 consensus_queue.cc:237] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.305089 1006 raft_consensus.cc:491] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:53.305161 1006 raft_consensus.cc:513] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:53.305437 1006 leader_election.cc:290] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:53.305719 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d751e2d40b94c3ebf4faaa9e621fc20" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9" is_pre_election: true
I20250628 02:04:53.305719 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d751e2d40b94c3ebf4faaa9e621fc20" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:53.305817 32119 raft_consensus.cc:2466] T 7d751e2d40b94c3ebf4faaa9e621fc20 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
I20250628 02:04:53.305817 32389 raft_consensus.cc:2466] T 7d751e2d40b94c3ebf4faaa9e621fc20 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
I20250628 02:04:53.306051 32188 leader_election.cc:304] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:53.306180 1006 raft_consensus.cc:2802] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:53.306231 1006 raft_consensus.cc:491] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:53.306275 1006 raft_consensus.cc:3058] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:53.306948 1006 raft_consensus.cc:513] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:04:53.307227 1006 leader_election.cc:290] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Requested vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:53.307535 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d751e2d40b94c3ebf4faaa9e621fc20" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9"
I20250628 02:04:53.307529 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d751e2d40b94c3ebf4faaa9e621fc20" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:53.307646 32119 raft_consensus.cc:3058] T 7d751e2d40b94c3ebf4faaa9e621fc20 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:53.307646 32389 raft_consensus.cc:3058] T 7d751e2d40b94c3ebf4faaa9e621fc20 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:53.308440 32119 raft_consensus.cc:2466] T 7d751e2d40b94c3ebf4faaa9e621fc20 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
I20250628 02:04:53.308440 32389 raft_consensus.cc:2466] T 7d751e2d40b94c3ebf4faaa9e621fc20 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
I20250628 02:04:53.308713 32187 leader_election.cc:304] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:53.308878 1006 raft_consensus.cc:2802] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:53.308951 1006 raft_consensus.cc:695] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [term 1 LEADER]: Becoming Leader. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Running, Role: LEADER
I20250628 02:04:53.309219 1006 consensus_queue.cc:237] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
W20250628 02:04:53.361114 32190 connection.cc:537] server connection from 127.0.0.1:53612 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250628 02:04:53.361160 32054 connection.cc:537] server connection from 127.0.0.1:59586 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250628 02:04:53.361155 32325 connection.cc:537] server connection from 127.0.0.1:36082 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250628 02:04:53.361202 32322 connection.cc:537] client connection to 127.30.198.62:42671 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250628 02:04:53.361284 32052 connection.cc:537] client connection to 127.30.198.62:42671 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250628 02:04:53.361807 32187 connection.cc:537] client connection to 127.30.198.62:42671 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250628 02:04:53.362032 32442 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.30.198.62:42671)
W20250628 02:04:53.362257 32442 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.198.62:42671)
W20250628 02:04:53.362613 32442 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.30.198.62:42671)
W20250628 02:04:53.362787 32442 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.30.198.62:42671)
I20250628 02:04:53.384832 1006 raft_consensus.cc:491] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:53.384915 1006 raft_consensus.cc:513] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.385224 1006 leader_election.cc:290] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733), cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:53.385517 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31216cf12a464b769be633b41088c417" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9" is_pre_election: true
I20250628 02:04:53.385525 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31216cf12a464b769be633b41088c417" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:53.385622 32119 raft_consensus.cc:2466] T 31216cf12a464b769be633b41088c417 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
I20250628 02:04:53.385634 32389 raft_consensus.cc:2466] T 31216cf12a464b769be633b41088c417 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
I20250628 02:04:53.385926 32187 leader_election.cc:304] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:53.386063 1006 raft_consensus.cc:2802] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:53.386124 1006 raft_consensus.cc:491] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:53.386150 1006 raft_consensus.cc:3058] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:53.386971 1006 raft_consensus.cc:513] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.387287 1006 leader_election.cc:290] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Requested vote from peers d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733), cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:53.387570 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31216cf12a464b769be633b41088c417" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:53.387645 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31216cf12a464b769be633b41088c417" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9"
I20250628 02:04:53.387705 32389 raft_consensus.cc:3058] T 31216cf12a464b769be633b41088c417 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:53.387776 32119 raft_consensus.cc:3058] T 31216cf12a464b769be633b41088c417 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:53.388410 32389 raft_consensus.cc:2466] T 31216cf12a464b769be633b41088c417 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
I20250628 02:04:53.388424 32119 raft_consensus.cc:2466] T 31216cf12a464b769be633b41088c417 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
I20250628 02:04:53.388656 32187 leader_election.cc:304] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:53.388794 1006 raft_consensus.cc:2802] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:53.388872 1006 raft_consensus.cc:695] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [term 1 LEADER]: Becoming Leader. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Running, Role: LEADER
I20250628 02:04:53.389019 1006 consensus_queue.cc:237] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:53.702075 1171 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:53.702190 1171 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:53.702513 1171 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:53.703445 1032 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:53.703522 1033 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:53.703576 1033 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:53.703716 1171 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250628 02:04:53.713256 1006 consensus_queue.cc:1035] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:53.716686 1006 consensus_queue.cc:1035] T 7d751e2d40b94c3ebf4faaa9e621fc20 P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:53.780627 1006 consensus_queue.cc:1035] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:53.784065 1177 consensus_queue.cc:1035] T 31216cf12a464b769be633b41088c417 P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:53.828536 1177 consensus_queue.cc:1035] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:53.832923 1166 consensus_queue.cc:1035] T 70e6b67d5b87462a96838d8229f0e13a P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:54.453861 1188 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:54.453991 1188 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:54.454363 1188 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:54.455380 1032 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:54.455478 1033 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:54.455560 1033 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:54.455714 1188 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250628 02:04:55.130461 1196 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:55.130564 1196 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:55.130931 1196 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:55.132119 1032 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:55.132254 1033 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:55.132323 1033 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:55.132458 1196 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250628 02:04:55.287367 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.62:42671
--webserver_interface=127.30.198.62
--webserver_port=40695
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250628 02:04:55.380015 1200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:04:55.380365 1200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:04:55.380448 1200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:04:55.383386 1200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 02:04:55.383437 1200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:04:55.383466 1200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 02:04:55.383500 1200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 02:04:55.386560 1200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.62:42671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.198.62
--webserver_port=40695
--never_fsync=true
--heap_profile_path=/tmp/kudu.1200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:04:55.387454 1200 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:04:55.388401 1200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 02:04:55.393890 1211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:55.394186 1212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:55.394448 1200 server_base.cc:1048] running on GCE node
W20250628 02:04:55.395524 1214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:55.395979 1200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:04:55.396752 1200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:04:55.397897 1200 hybrid_clock.cc:648] HybridClock initialized: now 1751076295397884 us; error 25 us; skew 500 ppm
I20250628 02:04:55.399506 1200 webserver.cc:469] Webserver started at http://127.30.198.62:40695/ using document root <none> and password file <none>
I20250628 02:04:55.399996 1200 fs_manager.cc:362] Metadata directory not provided
I20250628 02:04:55.400053 1200 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:04:55.402846 1200 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:55.404462 1220 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:55.405264 1200 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:55.405373 1200 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
uuid: "e9fd188e41da4ec0b493c144accb0344"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:55.405659 1200 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:04:55.413232 1200 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:04:55.413695 1200 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:04:55.413828 1200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:04:55.419601 1200 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.62:42671
I20250628 02:04:55.419615 1272 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.62:42671 every 8 connection(s)
I20250628 02:04:55.420451 1200 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/info.pb
I20250628 02:04:55.422138 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 1200
I20250628 02:04:55.423739 1273 sys_catalog.cc:263] Verifying existing consensus state
I20250628 02:04:55.425367 1273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Bootstrap starting.
I20250628 02:04:55.432492 1273 log.cc:826] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Log is configured to *not* fsync() on all Append() calls
I20250628 02:04:55.457615 1273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Bootstrap replayed 1/1 log segments. Stats: ops{read=56 overwritten=0 applied=56 ignored=0} inserts{seen=34 ignored=0} mutations{seen=39 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 02:04:55.457948 1273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Bootstrap complete.
I20250628 02:04:55.460033 1273 raft_consensus.cc:357] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:55.460400 1273 raft_consensus.cc:738] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fd188e41da4ec0b493c144accb0344, State: Initialized, Role: FOLLOWER
I20250628 02:04:55.460757 1273 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 56, Last appended: 8.56, Last appended by leader: 56, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:55.461071 1277 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:55.461171 1277 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:55.461548 1273 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 02:04:55.466379 1291 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:55.466518 1291 catalog_manager.cc:1554] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: loading cluster ID for follower catalog manager: success
I20250628 02:04:55.473251 1291 catalog_manager.cc:1576] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: acquiring CA information for follower catalog manager: success
I20250628 02:04:55.474736 1291 catalog_manager.cc:1604] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250628 02:04:55.477669 1200 master_runner.cc:186] Error getting master registration for 127.30.198.60:40321: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111),
I20250628 02:04:55.479445 1200 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:04:55.839761 32166 heartbeater.cc:344] Connected to a master server at 127.30.198.62:42671
I20250628 02:04:55.839872 32166 heartbeater.cc:499] Master 127.30.198.62:42671 was elected leader, sending a full tablet report...
I20250628 02:04:55.841588 32436 heartbeater.cc:344] Connected to a master server at 127.30.198.62:42671
I20250628 02:04:55.841687 32436 heartbeater.cc:499] Master 127.30.198.62:42671 was elected leader, sending a full tablet report...
I20250628 02:04:55.841646 1237 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cb3d5d42c47546258deed617114e71f9" instance_seqno: 1751076273823804) as {username='slave'} at 127.30.198.1:55295; Asking this server to re-register.
I20250628 02:04:55.842131 32166 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:55.842691 1237 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" instance_seqno: 1751076274129070) as {username='slave'} at 127.30.198.3:44041; Asking this server to re-register.
I20250628 02:04:55.842998 32436 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:55.843263 1236 ts_manager.cc:194] Registered new tserver with Master: cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:55.843729 1237 ts_manager.cc:194] Registered new tserver with Master: d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:55.867412 32301 heartbeater.cc:344] Connected to a master server at 127.30.198.62:42671
I20250628 02:04:55.868507 1237 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" instance_seqno: 1751076273975274) as {username='slave'} at 127.30.198.2:54697; Asking this server to re-register.
I20250628 02:04:55.868889 32301 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:55.869570 1237 ts_manager.cc:194] Registered new tserver with Master: b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:55.990509 1303 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:55.990667 1303 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:55.991735 1303 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
I20250628 02:04:55.992055 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 9 candidate_status { last_received { term: 8 index: 56 } } ignore_live_leader: false dest_uuid: "36911adab67246cfbfaf77ef5221fc92" is_pre_election: true
I20250628 02:04:55.992178 1058 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fd188e41da4ec0b493c144accb0344 in term 8.
W20250628 02:04:55.992206 1222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:55.992496 1223 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344; no voters:
I20250628 02:04:55.992699 1303 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250628 02:04:55.992764 1303 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:55.992815 1303 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 8 FOLLOWER]: Advancing to term 9
W20250628 02:04:55.992877 1222 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:55.994315 1303 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:55.994607 1303 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 9 election: Requested vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
I20250628 02:04:55.994858 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 9 candidate_status { last_received { term: 8 index: 56 } } ignore_live_leader: false dest_uuid: "36911adab67246cfbfaf77ef5221fc92"
I20250628 02:04:55.994956 1058 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 8 FOLLOWER]: Advancing to term 9
W20250628 02:04:55.995735 1222 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:55.995945 1058 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fd188e41da4ec0b493c144accb0344 in term 9.
I20250628 02:04:55.996182 1223 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344; no voters: 492448828d9a404fa245390eba0d56fb
I20250628 02:04:55.996357 1303 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 9 FOLLOWER]: Leader election won for term 9
I20250628 02:04:55.996483 1303 raft_consensus.cc:695] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 9 LEADER]: Becoming Leader. State: Replica: e9fd188e41da4ec0b493c144accb0344, State: Running, Role: LEADER
I20250628 02:04:55.996749 1303 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 8.56, Last appended by leader: 56, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:55.998214 1306 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fd188e41da4ec0b493c144accb0344. Latest consensus state: current_term: 9 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:55.998313 1306 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:55.998569 1308 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 02:04:56.000048 1308 catalog_manager.cc:671] Loaded metadata for table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]
I20250628 02:04:56.000371 1308 catalog_manager.cc:671] Loaded metadata for table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]
I20250628 02:04:56.000623 1308 catalog_manager.cc:671] Loaded metadata for table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]
I20250628 02:04:56.000874 1308 catalog_manager.cc:671] Loaded metadata for table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]
I20250628 02:04:56.001103 1308 catalog_manager.cc:671] Loaded metadata for table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]
I20250628 02:04:56.001312 1308 catalog_manager.cc:671] Loaded metadata for table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]
I20250628 02:04:56.001533 1308 catalog_manager.cc:671] Loaded metadata for table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]
I20250628 02:04:56.002656 1308 tablet_loader.cc:96] loaded metadata for tablet 31216cf12a464b769be633b41088c417 (table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c])
I20250628 02:04:56.002905 1308 tablet_loader.cc:96] loaded metadata for tablet 47667f26ca1e446dbd8e31d13410016c (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3])
I20250628 02:04:56.003285 1308 tablet_loader.cc:96] loaded metadata for tablet 70e6b67d5b87462a96838d8229f0e13a (table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003])
I20250628 02:04:56.003463 1308 tablet_loader.cc:96] loaded metadata for tablet 73d2f50340b14bd08abd193108c40b47 (table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc])
I20250628 02:04:56.003643 1308 tablet_loader.cc:96] loaded metadata for tablet 7d751e2d40b94c3ebf4faaa9e621fc20 (table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0])
I20250628 02:04:56.004330 1308 tablet_loader.cc:96] loaded metadata for tablet bc437af4ee094529ae0078641d819bbc (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e])
I20250628 02:04:56.004739 1308 tablet_loader.cc:96] loaded metadata for tablet e6af82fc366e4121967e834148998420 (table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb])
I20250628 02:04:56.004853 1308 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 02:04:56.005409 1308 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:56.005463 1308 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 02:04:56.006215 1308 catalog_manager.cc:1506] Loading token signing keys...
I20250628 02:04:56.006640 1308 catalog_manager.cc:5966] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Loaded TSK: 0
I20250628 02:04:56.007257 1308 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 02:04:56.007766 1308 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250628 02:04:56.008242 1308 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 19
I20250628 02:04:56.071332 1237 master_service.cc:948] ReplaceTablet: received request to replace tablet 47667f26ca1e446dbd8e31d13410016c from {username='slave'} at 127.0.0.1:41898
I20250628 02:04:56.071616 1235 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41898:
name: "default.Table_bed4f3bffa6d40138ec95e58cc15fba0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:56.071835 1230 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41898:
name: "default.Table_7eb9297b9cef4c8e82bbdb793ba152cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:56.071902 1234 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41898:
name: "default.Table_0f9f68e3f7dd46c29954d59183c6b66f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:56.072152 1233 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41898:
name: "default.Table_6296f6b4fab44283a0eb5a6ee76c4306"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:56.072379 1232 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41898:
name: "default.Table_57afb1ada3224fe1a269b48cdb5e87ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:56.072589 1231 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41898:
name: "default.Table_4687199de5174487b23a117d7fe19211"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:56.072818 1228 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41898:
table { table_name: "default.Table_d40b0217227a48fc9431fb713c4ed6be" } modify_external_catalogs: true
I20250628 02:04:56.072914 1228 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41898:
table { table_name: "default.Table_d40b0217227a48fc9431fb713c4ed6be" } modify_external_catalogs: true
I20250628 02:04:56.071364 1236 master_service.cc:948] ReplaceTablet: received request to replace tablet ac08335565ca46638e022c4588a592f0 from {username='slave'} at 127.0.0.1:41898
I20250628 02:04:56.073383 1236 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41898:
name: "default.Table_51147aa1de1d4fac95229bd0283919f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:56.073446 1229 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41898:
table { table_name: "default.Table_48852943c5c64ddfbde3e614d034c8d6" } modify_external_catalogs: true
I20250628 02:04:56.073539 1229 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41898:
table { table_name: "default.Table_48852943c5c64ddfbde3e614d034c8d6" } modify_external_catalogs: true
I20250628 02:04:56.074666 1058 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Refusing update from remote peer e9fd188e41da4ec0b493c144accb0344: Log matching property violated. Preceding OpId in replica: term: 8 index: 56. Preceding OpId from leader: term: 9 index: 58. (index mismatch)
I20250628 02:04:56.075079 1306 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
W20250628 02:04:56.075268 1222 consensus_peers.cc:489] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 -> Peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Couldn't send request to peer 492448828d9a404fa245390eba0d56fb. Status: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20250628 02:04:56.077806 1290 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 5 out of 10 received events: NotificationEvent(eventId=20, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_ccdc9aecb2d34778937160fc3d5cb4cc, message={"server":"","servicePrincipal":"","db":"default","table":"table_ccdc9aecb2d34778937160fc3d5cb4cc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ccdc9aecb2d34778937160fc3d5cb4cc\",\"kudu.table_id\":\"ae7629d68a7d4fdd9ed00f4b1b046cd3\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_0ba2136e171748f0ac70f94ad2378a15, message={"server":"","servicePrincipal":"","db":"default","table":"table_0ba2136e171748f0ac70f94ad2378a15","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0ba2136e171748f0ac70f94ad2378a15\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_0ba2136e171748f0ac70f94ad2378a15\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0ba2136e171748f0ac70f94ad2378a15\",\"kudu.table_id\":\"56f325f0d2044b03bda5136a7f717212\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1751076275, eventType=ALTER_TABLE, dbName=default, tableName=table_2ffede105ab7483388016f792547fd7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_25f7e6e53154401aa745623d0a9238e5","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_25f7e6e53154401aa745623d0a9238e5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_25f7e6e53154401aa745623d0a9238e5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_25f7e6e53154401aa745623d0a9238e5\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_2ffede105ab7483388016f792547fd7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_2ffede105ab7483388016f792547fd7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2ffede105ab7483388016f792547fd7f\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_48b5e32835b14bc489b2a0c3737240a3, message={"server":"","servicePrincipal":"","db":"default","table":"table_48b5e32835b14bc489b2a0c3737240a3","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_48b5e32835b14bc489b2a0c3737240a3\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_48b5e32835b14bc489b2a0c3737240a3\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_48b5e32835b14bc489b2a0c3737240a3\",\"kudu.table_id\":\"92619a7eecd54094bd8c3e908c2314e3\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_51acb52242224cffa3ce3ea99517351a, message={"server":"","servicePrincipal":"","db":"default","table":"table_51acb52242224cffa3ce3ea99517351a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_51acb52242224cffa3ce3ea99517351a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_51acb52242224cffa3ce3ea99517351a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_51acb52242224cffa3ce3ea99517351a\",\"kudu.table_id\":\"582401feb808444b885f2532faac36e0\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250628 02:04:56.077965 1309 mvcc.cc:204] Tried to move back new op lower bound from 7172408508717850624 to 7172408508404740096. Current Snapshot: MvccSnapshot[applied={T|T < 7172408508717850624}]
I20250628 02:04:56.078258 1303 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:56.078370 1303 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:56.078512 1310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fd188e41da4ec0b493c144accb0344. Latest consensus state: current_term: 9 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:56.078668 1310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:56.079411 1310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "e9fd188e41da4ec0b493c144accb0344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:56.079507 1310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:56.080063 1237 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Sending DeleteTablet for 3 replicas of tablet 47667f26ca1e446dbd8e31d13410016c
@ 0x7fd10eae7dcd google::LogMessage::Fail() at ??:0
I20250628 02:04:56.081296 1237 catalog_manager.cc:6652] ReplaceTablet: tablet 47667f26ca1e446dbd8e31d13410016c deleted and replaced by tablet f1a4e82fb4204029b5675791641b0629
I20250628 02:04:56.081527 1237 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41898:
table { table_name: "default.Table_25f7e6e53154401aa745623d0a9238e5" } new_table_name: "default.Table_2ffede105ab7483388016f792547fd7f" modify_external_catalogs: true
@ 0x7fd10eaebb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd10eae77cc google::LogMessage::Flush() at ??:0
@ 0x7fd10eae7a11 google::LogMessage::~LogMessage() at ??:0
I20250628 02:04:56.086917 32229 tablet_service.cc:1515] Processing DeleteTablet for tablet 47667f26ca1e446dbd8e31d13410016c with delete_type TABLET_DATA_DELETED (replaced by tablet f1a4e82fb4204029b5675791641b0629) from {username='slave'} at 127.0.0.1:53614
I20250628 02:04:56.087565 32229 tablet_service.cc:1468] Processing CreateTablet for tablet f1a4e82fb4204029b5675791641b0629 (DEFAULT_TABLE table=default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:56.087929 32229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a4e82fb4204029b5675791641b0629. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:56.087568 1322 tablet_replica.cc:331] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:56.088443 1322 raft_consensus.cc:2241] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [term 1 LEADER]: Raft consensus shutting down.
I20250628 02:04:56.088599 1322 raft_consensus.cc:2270] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:56.088934 1322 ts_tablet_manager.cc:1905] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fd113a80c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250628 02:04:56.090616 1322 ts_tablet_manager.cc:1918] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:56.090679 1322 log.cc:1199] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/47667f26ca1e446dbd8e31d13410016c
I20250628 02:04:56.090870 1322 ts_tablet_manager.cc:1939] T 47667f26ca1e446dbd8e31d13410016c P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
I20250628 02:04:56.091449 1224 catalog_manager.cc:4928] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697): tablet 47667f26ca1e446dbd8e31d13410016c (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]) successfully deleted
I20250628 02:04:56.091712 32099 tablet_service.cc:1515] Processing DeleteTablet for tablet 47667f26ca1e446dbd8e31d13410016c with delete_type TABLET_DATA_DELETED (replaced by tablet f1a4e82fb4204029b5675791641b0629) from {username='slave'} at 127.0.0.1:59590
I20250628 02:04:56.091910 32097 tablet_service.cc:1468] Processing CreateTablet for tablet f1a4e82fb4204029b5675791641b0629 (DEFAULT_TABLE table=default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:56.092173 32097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a4e82fb4204029b5675791641b0629. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:56.092495 1323 tablet_bootstrap.cc:492] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:56.093569 1324 tablet_replica.cc:331] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:56.093658 1324 raft_consensus.cc:2241] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:56.093744 1324 raft_consensus.cc:2270] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:56.093990 1324 ts_tablet_manager.cc:1905] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:56.094763 32365 tablet_service.cc:1515] Processing DeleteTablet for tablet 47667f26ca1e446dbd8e31d13410016c with delete_type TABLET_DATA_DELETED (replaced by tablet f1a4e82fb4204029b5675791641b0629) from {username='slave'} at 127.0.0.1:36094
I20250628 02:04:56.094964 32366 tablet_service.cc:1468] Processing CreateTablet for tablet f1a4e82fb4204029b5675791641b0629 (DEFAULT_TABLE table=default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:56.095233 32366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a4e82fb4204029b5675791641b0629. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:56.095508 1324 ts_tablet_manager.cc:1918] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:56.095552 1324 log.cc:1199] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/47667f26ca1e446dbd8e31d13410016c
I20250628 02:04:56.095749 1324 ts_tablet_manager.cc:1939] T 47667f26ca1e446dbd8e31d13410016c P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:04:56.096252 1221 catalog_manager.cc:4928] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393): tablet 47667f26ca1e446dbd8e31d13410016c (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]) successfully deleted
I20250628 02:04:56.096555 1323 tablet_bootstrap.cc:654] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
@ 0x7fd113a7fba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250628 02:04:56.097283 1325 tablet_bootstrap.cc:492] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:56.098049 1325 tablet_bootstrap.cc:654] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:56.098038 1329 tablet_bootstrap.cc:492] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:56.098505 1327 tablet_replica.cc:331] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:56.098590 1327 raft_consensus.cc:2241] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:56.098656 1327 raft_consensus.cc:2270] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:56.098884 1327 ts_tablet_manager.cc:1905] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:56.099153 1329 tablet_bootstrap.cc:654] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
@ 0x7fd113a7f30b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250628 02:04:56.100193 1323 tablet_bootstrap.cc:492] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:56.100235 1325 tablet_bootstrap.cc:492] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:56.100265 1323 ts_tablet_manager.cc:1397] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250628 02:04:56.100307 1325 ts_tablet_manager.cc:1397] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250628 02:04:56.100441 1327 ts_tablet_manager.cc:1918] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:56.100497 1327 log.cc:1199] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/47667f26ca1e446dbd8e31d13410016c
I20250628 02:04:56.100735 1323 raft_consensus.cc:357] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:56.100841 1323 raft_consensus.cc:383] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:56.100873 1323 raft_consensus.cc:738] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:56.100821 1325 raft_consensus.cc:357] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:56.100926 1325 raft_consensus.cc:383] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:56.100963 1325 raft_consensus.cc:738] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:56.101013 1327 ts_tablet_manager.cc:1939] T 47667f26ca1e446dbd8e31d13410016c P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:04:56.101013 1323 consensus_queue.cc:260] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:56.101289 1323 ts_tablet_manager.cc:1428] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250628 02:04:56.101102 1325 consensus_queue.cc:260] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:56.101814 1222 catalog_manager.cc:4928] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733): tablet 47667f26ca1e446dbd8e31d13410016c (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]) successfully deleted
I20250628 02:04:56.101990 1325 ts_tablet_manager.cc:1428] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250628 02:04:56.104565 1329 tablet_bootstrap.cc:492] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
@ 0x7fd113a84a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250628 02:04:56.104632 1329 ts_tablet_manager.cc:1397] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250628 02:04:56.104995 1329 raft_consensus.cc:357] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:56.105127 1329 raft_consensus.cc:383] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:56.105175 1329 raft_consensus.cc:738] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
@ 0x55b2db31e752 std::function<>::operator()() at ??:0
I20250628 02:04:56.105319 1329 consensus_queue.cc:260] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:56.105589 1329 ts_tablet_manager.cc:1428] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
@ 0x7fd10ff390d1 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd1107286db start_thread at ??:0
@ 0x7fd10dc9e71f clone at ??:0
I20250628 02:04:56.130211 1006 raft_consensus.cc:491] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:56.130280 1006 raft_consensus.cc:513] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:56.130591 1006 leader_election.cc:290] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733), cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:56.130895 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1a4e82fb4204029b5675791641b0629" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:56.130951 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1a4e82fb4204029b5675791641b0629" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9" is_pre_election: true
I20250628 02:04:56.131044 32389 raft_consensus.cc:2466] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
I20250628 02:04:56.131080 32119 raft_consensus.cc:2466] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
I20250628 02:04:56.131455 32187 leader_election.cc:304] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:56.131651 1006 raft_consensus.cc:2802] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:56.131722 1006 raft_consensus.cc:491] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:56.131765 1006 raft_consensus.cc:3058] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:56.132431 1006 raft_consensus.cc:513] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:56.132673 1006 leader_election.cc:290] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Requested vote from peers d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733), cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:56.132915 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1a4e82fb4204029b5675791641b0629" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:56.133028 32389 raft_consensus.cc:3058] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:56.133047 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1a4e82fb4204029b5675791641b0629" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9"
I20250628 02:04:56.133179 32119 raft_consensus.cc:3058] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:56.133873 32389 raft_consensus.cc:2466] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
I20250628 02:04:56.133996 32119 raft_consensus.cc:2466] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
I20250628 02:04:56.134265 32187 leader_election.cc:304] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:56.134418 1006 raft_consensus.cc:2802] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:56.134539 1006 raft_consensus.cc:695] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [term 1 LEADER]: Becoming Leader. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Running, Role: LEADER
I20250628 02:04:56.134632 1006 consensus_queue.cc:237] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
W20250628 02:04:56.245587 1301 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:56.245094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.62:42671 (local address 127.30.198.2:53157)
0628 02:04:56.245440 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:56.245485 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":246}
W20250628 02:04:56.245723 32442 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.30.198.62:42671)
W20250628 02:04:56.245994 32442 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.198.62:42671)
W20250628 02:04:56.246026 1296 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:56.245243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.62:42671 (local address 127.30.198.1:52825)
0628 02:04:56.245887 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:56.245939 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":549}
W20250628 02:04:56.246081 32442 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.30.198.62:42671)
W20250628 02:04:56.246641 32442 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.30.198.62:42671)
I20250628 02:04:56.582074 1342 consensus_queue.cc:1035] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:56.585146 1349 consensus_queue.cc:1035] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:56.586033 1352 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:56.586164 1352 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:56.586529 1352 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:56.588627 1032 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:56.589038 1033 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:56.589103 1033 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:56.589248 1352 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250628 02:04:57.228678 1361 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:57.228790 1361 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:57.229136 1361 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:57.230018 1032 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:57.230103 1033 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:57.230159 1033 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:57.230273 1361 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250628 02:04:57.446627 1017 master_runner.cc:186] Error getting master registration for 127.30.198.62:42671: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111),
I20250628 02:04:57.446722 1017 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:04:57.816524 1371 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:57.816627 1371 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:57.816995 1371 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:57.818003 1032 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:57.818149 1033 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:04:57.818218 1033 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92; no voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:57.818331 1371 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250628 02:04:58.327929 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.60:40321
--webserver_interface=127.30.198.60
--webserver_port=35681
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250628 02:04:58.420826 1380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:04:58.421037 1380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:04:58.421087 1380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:04:58.424050 1380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 02:04:58.424119 1380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:04:58.424152 1380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 02:04:58.424175 1380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 02:04:58.427496 1380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.60:40321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.198.60
--webserver_port=35681
--never_fsync=true
--heap_profile_path=/tmp/kudu.1380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:04:58.428356 1380 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:04:58.429154 1380 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 02:04:58.434540 1388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:04:58.434564 1386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:58.434721 1380 server_base.cc:1048] running on GCE node
W20250628 02:04:58.434543 1385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:04:58.435232 1380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:04:58.435946 1380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:04:58.437093 1380 hybrid_clock.cc:648] HybridClock initialized: now 1751076298437077 us; error 31 us; skew 500 ppm
I20250628 02:04:58.441872 1380 webserver.cc:469] Webserver started at http://127.30.198.60:35681/ using document root <none> and password file <none>
I20250628 02:04:58.442355 1380 fs_manager.cc:362] Metadata directory not provided
I20250628 02:04:58.442409 1380 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:04:58.445390 1380 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250628 02:04:58.446885 1394 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:04:58.447824 1380 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 02:04:58.447928 1380 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
uuid: "492448828d9a404fa245390eba0d56fb"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:04:58.448230 1380 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:04:58.474061 1380 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:04:58.474516 1380 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:04:58.474650 1380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:04:58.480378 1380 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.60:40321
I20250628 02:04:58.480387 1447 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.60:40321 every 8 connection(s)
I20250628 02:04:58.481192 1380 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/info.pb
I20250628 02:04:58.482724 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 1380
I20250628 02:04:58.484875 1448 sys_catalog.cc:263] Verifying existing consensus state
I20250628 02:04:58.486557 1448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Bootstrap starting.
I20250628 02:04:58.493234 1448 log.cc:826] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Log is configured to *not* fsync() on all Append() calls
I20250628 02:04:58.515954 1448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=50 overwritten=0 applied=50 ignored=0} inserts{seen=31 ignored=0} mutations{seen=33 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 02:04:58.516214 1448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Bootstrap complete.
I20250628 02:04:58.518306 1448 raft_consensus.cc:357] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:58.518621 1448 raft_consensus.cc:738] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448828d9a404fa245390eba0d56fb, State: Initialized, Role: FOLLOWER
I20250628 02:04:58.518942 1448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 50, Last appended: 7.50, Last appended by leader: 50, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:58.519275 1452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:58.519404 1452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:58.519858 1448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: configured and running, proceeding with master startup.
I20250628 02:04:58.523368 1380 master_runner.cc:186] Error getting master registration for 127.30.198.62:42671: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111),
I20250628 02:04:58.525115 1380 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:04:58.525326 1466 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:58.525372 1466 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: loading cluster ID for follower catalog manager: success
I20250628 02:04:58.527194 1466 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: acquiring CA information for follower catalog manager: success
I20250628 02:04:58.528455 1466 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250628 02:04:58.588078 32300 heartbeater.cc:344] Connected to a master server at 127.30.198.60:40321
I20250628 02:04:58.589883 1411 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" instance_seqno: 1751076273975274) as {username='slave'} at 127.30.198.2:52237; Asking this server to re-register.
I20250628 02:04:58.590528 32300 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:58.592963 1411 ts_manager.cc:194] Registered new tserver with Master: b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:58.593864 32435 heartbeater.cc:344] Connected to a master server at 127.30.198.60:40321
I20250628 02:04:58.594780 1411 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" instance_seqno: 1751076274129070) as {username='slave'} at 127.30.198.3:38625; Asking this server to re-register.
I20250628 02:04:58.595080 32165 heartbeater.cc:344] Connected to a master server at 127.30.198.60:40321
I20250628 02:04:58.595130 32435 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:58.595916 1410 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cb3d5d42c47546258deed617114e71f9" instance_seqno: 1751076273823804) as {username='slave'} at 127.30.198.1:56873; Asking this server to re-register.
I20250628 02:04:58.595989 1411 ts_manager.cc:194] Registered new tserver with Master: d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:04:58.596222 32165 heartbeater.cc:461] Registering TS with master...
I20250628 02:04:58.596889 1411 ts_manager.cc:194] Registered new tserver with Master: cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:58.992271 1452 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:58.992447 1452 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:58.993492 1452 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
W20250628 02:04:58.993917 1395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:58.994634 1395 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
I20250628 02:04:58.996564 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492448828d9a404fa245390eba0d56fb" candidate_term: 8 candidate_status { last_received { term: 7 index: 50 } } ignore_live_leader: false dest_uuid: "36911adab67246cfbfaf77ef5221fc92" is_pre_election: true
I20250628 02:04:58.996686 1058 raft_consensus.cc:2366] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 492448828d9a404fa245390eba0d56fb for earlier term 8. Current term is 9.
I20250628 02:04:58.997116 1397 leader_election.cc:400] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 8 pre-election: Vote denied by peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 492448828d9a404fa245390eba0d56fb for earlier term 8. Current term is 9.
I20250628 02:04:58.997164 1397 leader_election.cc:403] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20250628 02:04:58.997314 1452 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 7 FOLLOWER]: Advancing to term 9
I20250628 02:04:58.998935 1452 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 492448828d9a404fa245390eba0d56fb for earlier term 8. Current term is 9.
I20250628 02:04:59.282351 1480 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:59.282454 1480 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:59.282853 1480 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:04:59.283406 1032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20250628 02:04:59.284150 1032 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
I20250628 02:04:59.286284 1421 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36911adab67246cfbfaf77ef5221fc92" candidate_term: 10 candidate_status { last_received { term: 9 index: 59 } } ignore_live_leader: false dest_uuid: "492448828d9a404fa245390eba0d56fb" is_pre_election: true
I20250628 02:04:59.286521 1421 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36911adab67246cfbfaf77ef5221fc92 in term 9.
I20250628 02:04:59.286794 1033 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb; no voters: e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:59.286937 1480 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250628 02:04:59.286998 1480 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Starting leader election (detected failure of leader e9fd188e41da4ec0b493c144accb0344)
I20250628 02:04:59.287046 1480 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 9 FOLLOWER]: Advancing to term 10
I20250628 02:04:59.287868 1480 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:59.288147 1480 leader_election.cc:290] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 election: Requested vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
I20250628 02:04:59.288460 1421 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36911adab67246cfbfaf77ef5221fc92" candidate_term: 10 candidate_status { last_received { term: 9 index: 59 } } ignore_live_leader: false dest_uuid: "492448828d9a404fa245390eba0d56fb"
I20250628 02:04:59.288576 1421 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 9 FOLLOWER]: Advancing to term 10
W20250628 02:04:59.288954 1032 leader_election.cc:336] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
I20250628 02:04:59.289489 1421 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36911adab67246cfbfaf77ef5221fc92 in term 10.
I20250628 02:04:59.289724 1033 leader_election.cc:304] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb; no voters: e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:59.289876 1480 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 10 FOLLOWER]: Leader election won for term 10
I20250628 02:04:59.290053 1480 raft_consensus.cc:695] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [term 10 LEADER]: Becoming Leader. State: Replica: 36911adab67246cfbfaf77ef5221fc92, State: Running, Role: LEADER
I20250628 02:04:59.290369 1480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 9.59, Last appended by leader: 59, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:59.291215 1485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36911adab67246cfbfaf77ef5221fc92. Latest consensus state: current_term: 10 leader_uuid: "36911adab67246cfbfaf77ef5221fc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:59.291316 1485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: LEADER
I20250628 02:04:59.291603 1487 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 02:04:59.293120 1487 catalog_manager.cc:671] Loaded metadata for table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]
I20250628 02:04:59.293394 1487 catalog_manager.cc:671] Loaded metadata for table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]
I20250628 02:04:59.293603 1487 catalog_manager.cc:671] Loaded metadata for table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]
I20250628 02:04:59.293817 1487 catalog_manager.cc:671] Loaded metadata for table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]
I20250628 02:04:59.294016 1487 catalog_manager.cc:671] Loaded metadata for table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]
I20250628 02:04:59.294226 1487 catalog_manager.cc:671] Loaded metadata for table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]
I20250628 02:04:59.294437 1487 catalog_manager.cc:671] Loaded metadata for table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]
I20250628 02:04:59.295598 1487 tablet_loader.cc:96] loaded metadata for tablet 31216cf12a464b769be633b41088c417 (table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c])
I20250628 02:04:59.296205 1487 tablet_loader.cc:96] loaded metadata for tablet 70e6b67d5b87462a96838d8229f0e13a (table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003])
I20250628 02:04:59.296371 1487 tablet_loader.cc:96] loaded metadata for tablet 73d2f50340b14bd08abd193108c40b47 (table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc])
I20250628 02:04:59.296528 1487 tablet_loader.cc:96] loaded metadata for tablet 7d751e2d40b94c3ebf4faaa9e621fc20 (table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0])
I20250628 02:04:59.297080 1487 tablet_loader.cc:96] loaded metadata for tablet bc437af4ee094529ae0078641d819bbc (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e])
I20250628 02:04:59.297405 1487 tablet_loader.cc:96] loaded metadata for tablet e6af82fc366e4121967e834148998420 (table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb])
I20250628 02:04:59.297561 1487 tablet_loader.cc:96] loaded metadata for tablet f1a4e82fb4204029b5675791641b0629 (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3])
I20250628 02:04:59.297665 1487 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 02:04:59.298125 1487 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:04:59.298170 1487 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 02:04:59.298753 1487 catalog_manager.cc:1506] Loading token signing keys...
I20250628 02:04:59.299216 1487 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Loaded TSK: 0
I20250628 02:04:59.299896 1487 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 02:04:59.300370 1487 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250628 02:04:59.300812 1487 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 19
I20250628 02:04:59.325292 1047 master_service.cc:948] ReplaceTablet: received request to replace tablet e6af82fc366e4121967e834148998420 from {username='slave'} at 127.0.0.1:41356
I20250628 02:04:59.325907 1048 master_service.cc:948] ReplaceTablet: received request to replace tablet bc437af4ee094529ae0078641d819bbc from {username='slave'} at 127.0.0.1:41356
I20250628 02:04:59.325973 1041 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41356:
name: "default.Table_6296f6b4fab44283a0eb5a6ee76c4306"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:59.326579 1046 master_service.cc:948] ReplaceTablet: received request to replace tablet bc437af4ee094529ae0078641d819bbc from {username='slave'} at 127.0.0.1:41356
I20250628 02:04:59.326790 1045 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41356:
name: "default.Table_48b5e32835b14bc489b2a0c3737240a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:59.326897 1040 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41356:
name: "default.Table_0f9f68e3f7dd46c29954d59183c6b66f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:59.326972 1039 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41356:
name: "default.Table_bed4f3bffa6d40138ec95e58cc15fba0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:59.327243 1044 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41356:
table { table_name: "default.Table_51acb52242224cffa3ce3ea99517351a" } modify_external_catalogs: true
I20250628 02:04:59.327284 1042 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41356:
name: "default.Table_57afb1ada3224fe1a269b48cdb5e87ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:59.327358 1044 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41356:
table { table_name: "default.Table_51acb52242224cffa3ce3ea99517351a" } modify_external_catalogs: true
I20250628 02:04:59.327510 1043 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41356:
table { table_name: "default.Table_ccdc9aecb2d34778937160fc3d5cb4cc" } modify_external_catalogs: true
I20250628 02:04:59.327628 1043 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41356:
table { table_name: "default.Table_ccdc9aecb2d34778937160fc3d5cb4cc" } modify_external_catalogs: true
I20250628 02:04:59.328617 1421 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Refusing update from remote peer 36911adab67246cfbfaf77ef5221fc92: Log matching property violated. Preceding OpId in replica: term: 7 index: 50. Preceding OpId from leader: term: 10 index: 61. (index mismatch)
I20250628 02:04:59.329088 1485 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 50, Time since last communication: 0.000s
W20250628 02:04:59.329671 1032 consensus_peers.cc:489] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 -> Peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Couldn't send request to peer e9fd188e41da4ec0b493c144accb0344. Status: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20250628 02:04:59.332512 1103 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 5 out of 10 received events: NotificationEvent(eventId=20, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_ccdc9aecb2d34778937160fc3d5cb4cc, message={"server":"","servicePrincipal":"","db":"default","table":"table_ccdc9aecb2d34778937160fc3d5cb4cc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ccdc9aecb2d34778937160fc3d5cb4cc\",\"kudu.table_id\":\"ae7629d68a7d4fdd9ed00f4b1b046cd3\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_0ba2136e171748f0ac70f94ad2378a15, message={"server":"","servicePrincipal":"","db":"default","table":"table_0ba2136e171748f0ac70f94ad2378a15","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0ba2136e171748f0ac70f94ad2378a15\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_0ba2136e171748f0ac70f94ad2378a15\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0ba2136e171748f0ac70f94ad2378a15\",\"kudu.table_id\":\"56f325f0d2044b03bda5136a7f717212\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1751076275, eventType=ALTER_TABLE, dbName=default, tableName=table_2ffede105ab7483388016f792547fd7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_25f7e6e53154401aa745623d0a9238e5","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_25f7e6e53154401aa745623d0a9238e5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_25f7e6e53154401aa745623d0a9238e5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_25f7e6e53154401aa745623d0a9238e5\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_2ffede105ab7483388016f792547fd7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_2ffede105ab7483388016f792547fd7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2ffede105ab7483388016f792547fd7f\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_48b5e32835b14bc489b2a0c3737240a3, message={"server":"","servicePrincipal":"","db":"default","table":"table_48b5e32835b14bc489b2a0c3737240a3","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_48b5e32835b14bc489b2a0c3737240a3\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_48b5e32835b14bc489b2a0c3737240a3\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_48b5e32835b14bc489b2a0c3737240a3\",\"kudu.table_id\":\"92619a7eecd54094bd8c3e908c2314e3\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_51acb52242224cffa3ce3ea99517351a, message={"server":"","servicePrincipal":"","db":"default","table":"table_51acb52242224cffa3ce3ea99517351a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_51acb52242224cffa3ce3ea99517351a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_51acb52242224cffa3ce3ea99517351a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_51acb52242224cffa3ce3ea99517351a\",\"kudu.table_id\":\"582401feb808444b885f2532faac36e0\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250628 02:04:59.334122 1491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "36911adab67246cfbfaf77ef5221fc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:59.334273 1491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:59.334399 1491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "36911adab67246cfbfaf77ef5221fc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:59.334497 1491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:59.334445 1452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36911adab67246cfbfaf77ef5221fc92. Latest consensus state: current_term: 10 leader_uuid: "36911adab67246cfbfaf77ef5221fc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:59.334894 1452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:59.335870 1485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "36911adab67246cfbfaf77ef5221fc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:59.335964 1485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92 [sys.catalog]: This master's current role is: LEADER
@ 0x7f8095ac9dcd google::LogMessage::Fail() at ??:0
I20250628 02:04:59.336840 1452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "36911adab67246cfbfaf77ef5221fc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:04:59.336938 1452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:04:59.337767 1489 mvcc.cc:204] Tried to move back new op lower bound from 7172408522048245760 to 7172408496609091584. Current Snapshot: MvccSnapshot[applied={T|T < 7172408522048245760}]
@ 0x7f8095acdb93 google::LogMessage::SendToLog() at ??:0
I20250628 02:04:59.339144 1047 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Sending DeleteTablet for 3 replicas of tablet e6af82fc366e4121967e834148998420
I20250628 02:04:59.339380 1046 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41356:
name: "default.Table_4687199de5174487b23a117d7fe19211"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:59.339535 1048 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 36911adab67246cfbfaf77ef5221fc92: Sending DeleteTablet for 3 replicas of tablet bc437af4ee094529ae0078641d819bbc
I20250628 02:04:59.340076 1047 catalog_manager.cc:6652] ReplaceTablet: tablet e6af82fc366e4121967e834148998420 deleted and replaced by tablet bfff6a29b6c34a1291ee7c92793829f7
I20250628 02:04:59.340261 1047 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41356:
name: "default.Table_7eb9297b9cef4c8e82bbdb793ba152cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:04:59.340370 1048 catalog_manager.cc:6652] ReplaceTablet: tablet bc437af4ee094529ae0078641d819bbc deleted and replaced by tablet c8d15f43cc6c4adc85f5f1a532c9092b
I20250628 02:04:59.340533 1048 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41356:
table { table_name: "default.Table_48852943c5c64ddfbde3e614d034c8d6" } modify_external_catalogs: true
I20250628 02:04:59.340623 1048 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41356:
table { table_name: "default.Table_48852943c5c64ddfbde3e614d034c8d6" } modify_external_catalogs: true
@ 0x7f8095ac97cc google::LogMessage::Flush() at ??:0
@ 0x7f8095ac9a11 google::LogMessage::~LogMessage() at ??:0
I20250628 02:04:59.347476 32229 tablet_service.cc:1515] Processing DeleteTablet for tablet e6af82fc366e4121967e834148998420 with delete_type TABLET_DATA_DELETED (replaced by tablet bfff6a29b6c34a1291ee7c92793829f7) from {username='slave'} at 127.0.0.1:53630
I20250628 02:04:59.347728 32232 tablet_service.cc:1515] Processing DeleteTablet for tablet bc437af4ee094529ae0078641d819bbc with delete_type TABLET_DATA_DELETED (replaced by tablet c8d15f43cc6c4adc85f5f1a532c9092b) from {username='slave'} at 127.0.0.1:53630
I20250628 02:04:59.347817 32230 tablet_service.cc:1468] Processing CreateTablet for tablet c8d15f43cc6c4adc85f5f1a532c9092b (DEFAULT_TABLE table=default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:59.348006 32233 tablet_service.cc:1468] Processing CreateTablet for tablet bfff6a29b6c34a1291ee7c92793829f7 (DEFAULT_TABLE table=default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:59.348109 32230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d15f43cc6c4adc85f5f1a532c9092b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:59.348270 32233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfff6a29b6c34a1291ee7c92793829f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:59.350759 1504 tablet_replica.cc:331] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:59.350895 1506 tablet_bootstrap.cc:492] T c8d15f43cc6c4adc85f5f1a532c9092b P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:59.350917 1504 raft_consensus.cc:2241] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:59.351038 1504 raft_consensus.cc:2270] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f809aa62c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250628 02:04:59.351536 1504 ts_tablet_manager.cc:1905] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:59.351873 1506 tablet_bootstrap.cc:654] T c8d15f43cc6c4adc85f5f1a532c9092b P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:59.352365 32366 tablet_service.cc:1515] Processing DeleteTablet for tablet e6af82fc366e4121967e834148998420 with delete_type TABLET_DATA_DELETED (replaced by tablet bfff6a29b6c34a1291ee7c92793829f7) from {username='slave'} at 127.0.0.1:36098
I20250628 02:04:59.352550 32365 tablet_service.cc:1515] Processing DeleteTablet for tablet bc437af4ee094529ae0078641d819bbc with delete_type TABLET_DATA_DELETED (replaced by tablet c8d15f43cc6c4adc85f5f1a532c9092b) from {username='slave'} at 127.0.0.1:36098
I20250628 02:04:59.352802 1507 tablet_replica.cc:331] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:59.352835 32368 tablet_service.cc:1468] Processing CreateTablet for tablet c8d15f43cc6c4adc85f5f1a532c9092b (DEFAULT_TABLE table=default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:59.352941 1507 raft_consensus.cc:2241] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:59.353051 1507 raft_consensus.cc:2270] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:59.353210 32367 tablet_service.cc:1468] Processing CreateTablet for tablet bfff6a29b6c34a1291ee7c92793829f7 (DEFAULT_TABLE table=default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:59.353060 32368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d15f43cc6c4adc85f5f1a532c9092b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:59.353657 1504 ts_tablet_manager.cc:1918] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:59.353717 1504 log.cc:1199] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/e6af82fc366e4121967e834148998420
I20250628 02:04:59.353948 1504 ts_tablet_manager.cc:1939] T e6af82fc366e4121967e834148998420 P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
I20250628 02:04:59.353963 32099 tablet_service.cc:1515] Processing DeleteTablet for tablet bc437af4ee094529ae0078641d819bbc with delete_type TABLET_DATA_DELETED (replaced by tablet c8d15f43cc6c4adc85f5f1a532c9092b) from {username='slave'} at 127.0.0.1:59606
I20250628 02:04:59.353914 32098 tablet_service.cc:1468] Processing CreateTablet for tablet c8d15f43cc6c4adc85f5f1a532c9092b (DEFAULT_TABLE table=default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:59.354171 32097 tablet_service.cc:1515] Processing DeleteTablet for tablet e6af82fc366e4121967e834148998420 with delete_type TABLET_DATA_DELETED (replaced by tablet bfff6a29b6c34a1291ee7c92793829f7) from {username='slave'} at 127.0.0.1:59606
I20250628 02:04:59.354220 32098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d15f43cc6c4adc85f5f1a532c9092b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:59.354482 1504 tablet_replica.cc:331] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:04:59.353914 32096 tablet_service.cc:1468] Processing CreateTablet for tablet bfff6a29b6c34a1291ee7c92793829f7 (DEFAULT_TABLE table=default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:04:59.354578 1504 raft_consensus.cc:2241] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:59.354703 1504 raft_consensus.cc:2270] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:59.354779 32096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfff6a29b6c34a1291ee7c92793829f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:59.354997 1504 ts_tablet_manager.cc:1905] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:59.355702 32367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfff6a29b6c34a1291ee7c92793829f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:04:59.356029 1035 catalog_manager.cc:4928] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697): tablet e6af82fc366e4121967e834148998420 (table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]) successfully deleted
I20250628 02:04:59.354588 1506 tablet_bootstrap.cc:492] T c8d15f43cc6c4adc85f5f1a532c9092b P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:59.356824 1506 ts_tablet_manager.cc:1397] T c8d15f43cc6c4adc85f5f1a532c9092b P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250628 02:04:59.356864 1504 ts_tablet_manager.cc:1918] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:59.356920 1504 log.cc:1199] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/bc437af4ee094529ae0078641d819bbc
I20250628 02:04:59.357188 1504 ts_tablet_manager.cc:1939] T bc437af4ee094529ae0078641d819bbc P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
I20250628 02:04:59.357234 1510 tablet_bootstrap.cc:492] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:04:59.357313 1506 raft_consensus.cc:357] T c8d15f43cc6c4adc85f5f1a532c9092b P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } }
I20250628 02:04:59.357411 1506 raft_consensus.cc:383] T c8d15f43cc6c4adc85f5f1a532c9092b P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:59.357445 1506 raft_consensus.cc:738] T c8d15f43cc6c4adc85f5f1a532c9092b P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:59.357553 1506 consensus_queue.cc:260] T c8d15f43cc6c4adc85f5f1a532c9092b P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } }
I20250628 02:04:59.357743 1035 catalog_manager.cc:4928] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697): tablet bc437af4ee094529ae0078641d819bbc (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]) successfully deleted
I20250628 02:04:59.357810 1506 ts_tablet_manager.cc:1428] T c8d15f43cc6c4adc85f5f1a532c9092b P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250628 02:04:59.357956 1506 tablet_bootstrap.cc:492] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:04:59.358541 1510 tablet_bootstrap.cc:654] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:59.359162 1506 tablet_bootstrap.cc:654] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:59.359418 1511 tablet_bootstrap.cc:492] T c8d15f43cc6c4adc85f5f1a532c9092b P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:04:59.359906 1507 ts_tablet_manager.cc:1905] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:59.360589 1511 tablet_bootstrap.cc:654] T c8d15f43cc6c4adc85f5f1a532c9092b P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
@ 0x7f809aa61ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250628 02:04:59.361538 1506 tablet_bootstrap.cc:492] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:04:59.361610 1506 ts_tablet_manager.cc:1397] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250628 02:04:59.361918 1507 ts_tablet_manager.cc:1918] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:59.361979 1507 log.cc:1199] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/e6af82fc366e4121967e834148998420
I20250628 02:04:59.362056 1506 raft_consensus.cc:357] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:59.362165 1506 raft_consensus.cc:383] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:59.362201 1506 raft_consensus.cc:738] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:04:59.362254 1507 ts_tablet_manager.cc:1939] T e6af82fc366e4121967e834148998420 P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:04:59.362327 1506 consensus_queue.cc:260] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:59.362574 1506 ts_tablet_manager.cc:1428] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250628 02:04:59.362771 1507 tablet_replica.cc:331] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:04:59.362826 1033 catalog_manager.cc:4928] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733): tablet e6af82fc366e4121967e834148998420 (table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]) successfully deleted
I20250628 02:04:59.362874 1507 raft_consensus.cc:2241] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:04:59.362962 1507 raft_consensus.cc:2270] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:59.363287 1507 ts_tablet_manager.cc:1905] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:59.363322 1510 tablet_bootstrap.cc:492] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:59.363382 1510 ts_tablet_manager.cc:1397] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250628 02:04:59.363822 1510 raft_consensus.cc:357] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } }
I20250628 02:04:59.363904 1510 raft_consensus.cc:383] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:59.363940 1510 raft_consensus.cc:738] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:59.364040 1510 consensus_queue.cc:260] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } }
@ 0x7f809aa6130b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250628 02:04:59.364949 1507 ts_tablet_manager.cc:1918] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:59.365007 1507 log.cc:1199] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/bc437af4ee094529ae0078641d819bbc
I20250628 02:04:59.365226 1507 ts_tablet_manager.cc:1939] T bc437af4ee094529ae0078641d819bbc P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:04:59.365769 1509 tablet_replica.cc:331] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:59.365892 1509 raft_consensus.cc:2241] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Raft consensus shutting down.
I20250628 02:04:59.365952 1033 catalog_manager.cc:4928] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733): tablet bc437af4ee094529ae0078641d819bbc (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]) successfully deleted
I20250628 02:04:59.366086 1509 raft_consensus.cc:2270] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:59.366299 1510 ts_tablet_manager.cc:1428] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250628 02:04:59.366730 1509 ts_tablet_manager.cc:1905] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:04:59.366987 1510 tablet_bootstrap.cc:492] T bfff6a29b6c34a1291ee7c92793829f7 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
@ 0x7f809aa66a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250628 02:04:59.367908 1510 tablet_bootstrap.cc:654] T bfff6a29b6c34a1291ee7c92793829f7 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
@ 0x55e3825bc752 std::function<>::operator()() at ??:0
I20250628 02:04:59.368899 1511 tablet_bootstrap.cc:492] T c8d15f43cc6c4adc85f5f1a532c9092b P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:59.368952 1509 ts_tablet_manager.cc:1918] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:59.368963 1511 ts_tablet_manager.cc:1397] T c8d15f43cc6c4adc85f5f1a532c9092b P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250628 02:04:59.369021 1509 log.cc:1199] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/bc437af4ee094529ae0078641d819bbc
I20250628 02:04:59.369285 1509 ts_tablet_manager.cc:1939] T bc437af4ee094529ae0078641d819bbc P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:04:59.369464 1511 raft_consensus.cc:357] T c8d15f43cc6c4adc85f5f1a532c9092b P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } }
I20250628 02:04:59.369541 1511 raft_consensus.cc:383] T c8d15f43cc6c4adc85f5f1a532c9092b P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:59.369568 1511 raft_consensus.cc:738] T c8d15f43cc6c4adc85f5f1a532c9092b P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:59.369808 1032 catalog_manager.cc:4928] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393): tablet bc437af4ee094529ae0078641d819bbc (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]) successfully deleted
I20250628 02:04:59.369686 1511 consensus_queue.cc:260] T c8d15f43cc6c4adc85f5f1a532c9092b P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } }
I20250628 02:04:59.370272 1510 tablet_bootstrap.cc:492] T bfff6a29b6c34a1291ee7c92793829f7 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:04:59.370328 1510 ts_tablet_manager.cc:1397] T bfff6a29b6c34a1291ee7c92793829f7 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250628 02:04:59.370500 1511 ts_tablet_manager.cc:1428] T c8d15f43cc6c4adc85f5f1a532c9092b P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
@ 0x7f8096f1b0d1 kudu::Thread::SuperviseThread() at ??:0
I20250628 02:04:59.370775 1510 raft_consensus.cc:357] T bfff6a29b6c34a1291ee7c92793829f7 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:59.370848 1510 raft_consensus.cc:383] T bfff6a29b6c34a1291ee7c92793829f7 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:59.370867 1510 raft_consensus.cc:738] T bfff6a29b6c34a1291ee7c92793829f7 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:04:59.370967 1510 consensus_queue.cc:260] T bfff6a29b6c34a1291ee7c92793829f7 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:59.371201 1510 ts_tablet_manager.cc:1428] T bfff6a29b6c34a1291ee7c92793829f7 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 02:04:59.369750 1509 tablet_replica.cc:331] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:04:59.371430 1509 raft_consensus.cc:2241] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [term 1 LEADER]: Raft consensus shutting down.
I20250628 02:04:59.371588 1509 raft_consensus.cc:2270] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:04:59.371930 1509 ts_tablet_manager.cc:1905] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f809770a6db start_thread at ??:0
I20250628 02:04:59.373507 1509 ts_tablet_manager.cc:1918] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:04:59.373564 1509 log.cc:1199] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/e6af82fc366e4121967e834148998420
I20250628 02:04:59.373769 1509 ts_tablet_manager.cc:1939] T e6af82fc366e4121967e834148998420 P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:04:59.374228 1032 catalog_manager.cc:4928] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393): tablet e6af82fc366e4121967e834148998420 (table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]) successfully deleted
I20250628 02:04:59.374408 1511 tablet_bootstrap.cc:492] T bfff6a29b6c34a1291ee7c92793829f7 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
@ 0x7f8094c8071f clone at ??:0
I20250628 02:04:59.376149 1511 tablet_bootstrap.cc:654] T bfff6a29b6c34a1291ee7c92793829f7 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:04:59.377086 1511 tablet_bootstrap.cc:492] T bfff6a29b6c34a1291ee7c92793829f7 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:04:59.377161 1511 ts_tablet_manager.cc:1397] T bfff6a29b6c34a1291ee7c92793829f7 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250628 02:04:59.377619 1511 raft_consensus.cc:357] T bfff6a29b6c34a1291ee7c92793829f7 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:59.377686 1511 raft_consensus.cc:383] T bfff6a29b6c34a1291ee7c92793829f7 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:04:59.377707 1511 raft_consensus.cc:738] T bfff6a29b6c34a1291ee7c92793829f7 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:04:59.377802 1511 consensus_queue.cc:260] T bfff6a29b6c34a1291ee7c92793829f7 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:59.378448 1511 ts_tablet_manager.cc:1428] T bfff6a29b6c34a1291ee7c92793829f7 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250628 02:04:59.505813 1342 raft_consensus.cc:491] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:59.505931 1342 raft_consensus.cc:513] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:59.506377 1342 leader_election.cc:290] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733), cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:59.506798 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfff6a29b6c34a1291ee7c92793829f7" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9" is_pre_election: true
I20250628 02:04:59.506814 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfff6a29b6c34a1291ee7c92793829f7" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685" is_pre_election: true
I20250628 02:04:59.506928 32119 raft_consensus.cc:2466] T bfff6a29b6c34a1291ee7c92793829f7 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
I20250628 02:04:59.506964 32389 raft_consensus.cc:2466] T bfff6a29b6c34a1291ee7c92793829f7 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 0.
I20250628 02:04:59.507367 32187 leader_election.cc:304] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:59.507577 1342 raft_consensus.cc:2802] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:59.507638 1342 raft_consensus.cc:491] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:59.507704 1342 raft_consensus.cc:3058] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:59.508870 1342 raft_consensus.cc:513] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
I20250628 02:04:59.509251 1342 leader_election.cc:290] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Requested vote from peers d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733), cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:04:59.509522 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfff6a29b6c34a1291ee7c92793829f7" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9"
I20250628 02:04:59.509545 32389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfff6a29b6c34a1291ee7c92793829f7" candidate_uuid: "b9085abee1ab4c96a3716383fe8fe592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aa6633f9b6481bb29b00d602764685"
I20250628 02:04:59.509622 32119 raft_consensus.cc:3058] T bfff6a29b6c34a1291ee7c92793829f7 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:59.509657 32389 raft_consensus.cc:3058] T bfff6a29b6c34a1291ee7c92793829f7 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:59.510639 32119 raft_consensus.cc:2466] T bfff6a29b6c34a1291ee7c92793829f7 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
I20250628 02:04:59.510942 32187 leader_election.cc:304] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, cb3d5d42c47546258deed617114e71f9; no voters:
I20250628 02:04:59.510810 32389 raft_consensus.cc:2466] T bfff6a29b6c34a1291ee7c92793829f7 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9085abee1ab4c96a3716383fe8fe592 in term 1.
I20250628 02:04:59.511210 1342 raft_consensus.cc:2802] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:59.511384 1342 raft_consensus.cc:695] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [term 1 LEADER]: Becoming Leader. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Running, Role: LEADER
I20250628 02:04:59.511530 1342 consensus_queue.cc:237] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } }
W20250628 02:04:59.517395 1470 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:59.516739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.61:35059 (local address 127.30.198.2:41093)
0628 02:04:59.516878 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:59.516891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0628 02:04:59.516965 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0628 02:04:59.517285 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":77}
W20250628 02:04:59.517506 1472 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:59.516712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.61:35059 (local address 127.30.198.3:36075)
0628 02:04:59.516875 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:59.516888 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0628 02:04:59.516967 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0628 02:04:59.517409 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":89}
W20250628 02:04:59.517628 32444 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.30.198.61:35059)
W20250628 02:04:59.517839 32444 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.30.198.61:35059)
W20250628 02:04:59.517948 32444 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.30.198.61:35059)
W20250628 02:04:59.518049 32444 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.198.61:35059)
W20250628 02:04:59.518699 1474 negotiation.cc:337] Failed RPC negotiation. Trace:
0628 02:04:59.516835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.198.61:35059 (local address 127.30.198.1:52263)
0628 02:04:59.517037 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0628 02:04:59.517052 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0628 02:04:59.517121 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0628 02:04:59.518606 (+ 1485us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":93}
I20250628 02:04:59.669510 32673 raft_consensus.cc:491] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:04:59.669613 32673 raft_consensus.cc:513] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } }
I20250628 02:04:59.669991 32673 leader_election.cc:290] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:59.670287 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8d15f43cc6c4adc85f5f1a532c9092b" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9" is_pre_election: true
I20250628 02:04:59.670303 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8d15f43cc6c4adc85f5f1a532c9092b" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592" is_pre_election: true
I20250628 02:04:59.670401 32119 raft_consensus.cc:2466] T c8d15f43cc6c4adc85f5f1a532c9092b P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 0.
I20250628 02:04:59.670423 32254 raft_consensus.cc:2466] T c8d15f43cc6c4adc85f5f1a532c9092b P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 0.
I20250628 02:04:59.670632 32322 leader_election.cc:304] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb3d5d42c47546258deed617114e71f9, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:59.670782 32673 raft_consensus.cc:2802] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:04:59.670835 32673 raft_consensus.cc:491] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:04:59.670864 32673 raft_consensus.cc:3058] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:59.671586 32673 raft_consensus.cc:513] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } }
I20250628 02:04:59.671916 32673 leader_election.cc:290] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 election: Requested vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:04:59.672210 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8d15f43cc6c4adc85f5f1a532c9092b" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592"
I20250628 02:04:59.672300 32254 raft_consensus.cc:3058] T c8d15f43cc6c4adc85f5f1a532c9092b P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:59.672266 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8d15f43cc6c4adc85f5f1a532c9092b" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9"
I20250628 02:04:59.672384 32119 raft_consensus.cc:3058] T c8d15f43cc6c4adc85f5f1a532c9092b P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:04:59.672981 32254 raft_consensus.cc:2466] T c8d15f43cc6c4adc85f5f1a532c9092b P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 1.
I20250628 02:04:59.673120 32119 raft_consensus.cc:2466] T c8d15f43cc6c4adc85f5f1a532c9092b P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 1.
I20250628 02:04:59.673241 32325 leader_election.cc:304] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:04:59.673559 32673 raft_consensus.cc:2802] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:04:59.673720 32673 raft_consensus.cc:695] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [term 1 LEADER]: Becoming Leader. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Running, Role: LEADER
I20250628 02:04:59.673862 32673 consensus_queue.cc:237] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } } peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } }
I20250628 02:04:59.846450 1533 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 36911adab67246cfbfaf77ef5221fc92)
I20250628 02:04:59.846545 1533 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:04:59.846876 1533 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
W20250628 02:04:59.847986 1395 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:04:59.848073 1397 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:04:59.848165 1397 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 492448828d9a404fa245390eba0d56fb; no voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344
I20250628 02:04:59.848332 1533 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250628 02:04:59.922309 1342 consensus_queue.cc:1035] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:04:59.925526 1527 consensus_queue.cc:1035] T bfff6a29b6c34a1291ee7c92793829f7 P b9085abee1ab4c96a3716383fe8fe592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:05:00.160351 32673 consensus_queue.cc:1035] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:05:00.163285 1546 consensus_queue.cc:1035] T c8d15f43cc6c4adc85f5f1a532c9092b P d0aa6633f9b6481bb29b00d602764685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:05:00.583357 1549 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 36911adab67246cfbfaf77ef5221fc92)
I20250628 02:05:00.583451 1549 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:00.583770 1549 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
W20250628 02:05:00.584699 1395 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:05:00.584780 1397 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:00.584846 1397 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 492448828d9a404fa245390eba0d56fb; no voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344
I20250628 02:05:00.584960 1549 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250628 02:05:00.935580 31512 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu with pid 1380
I20250628 02:05:01.653829 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.60:40321
--webserver_interface=127.30.198.60
--webserver_port=35681
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250628 02:05:01.744217 1566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:05:01.744391 1566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:05:01.744436 1566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:05:01.747262 1566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 02:05:01.747325 1566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:05:01.747347 1566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 02:05:01.747392 1566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 02:05:01.750650 1566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.60:40321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.198.60
--webserver_port=35681
--never_fsync=true
--heap_profile_path=/tmp/kudu.1566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:05:01.751441 1566 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:05:01.752353 1566 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 02:05:01.757939 1577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:05:01.757921 1575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:05:01.757921 1574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:05:01.758255 1566 server_base.cc:1048] running on GCE node
I20250628 02:05:01.758678 1566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:05:01.759433 1566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:05:01.760564 1566 hybrid_clock.cc:648] HybridClock initialized: now 1751076301760542 us; error 34 us; skew 500 ppm
I20250628 02:05:01.762269 1566 webserver.cc:469] Webserver started at http://127.30.198.60:35681/ using document root <none> and password file <none>
I20250628 02:05:01.762740 1566 fs_manager.cc:362] Metadata directory not provided
I20250628 02:05:01.762796 1566 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:05:01.765687 1566 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20250628 02:05:01.767589 1583 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:05:01.768697 1566 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250628 02:05:01.768790 1566 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
uuid: "492448828d9a404fa245390eba0d56fb"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:05:01.769091 1566 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:05:01.782754 1566 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:05:01.783320 1566 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:05:01.783488 1566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:05:01.789566 1566 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.60:40321
I20250628 02:05:01.789577 1635 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.60:40321 every 8 connection(s)
I20250628 02:05:01.790491 1566 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-2/data/info.pb
I20250628 02:05:01.793704 1636 sys_catalog.cc:263] Verifying existing consensus state
I20250628 02:05:01.794984 1636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Bootstrap starting.
I20250628 02:05:01.798775 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 1566
I20250628 02:05:01.803891 1636 log.cc:826] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Log is configured to *not* fsync() on all Append() calls
I20250628 02:05:01.831846 1636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=63 overwritten=0 applied=63 ignored=0} inserts{seen=37 ignored=0} mutations{seen=45 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 02:05:01.832149 1636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Bootstrap complete.
I20250628 02:05:01.834201 1636 raft_consensus.cc:357] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:01.834475 1636 raft_consensus.cc:738] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448828d9a404fa245390eba0d56fb, State: Initialized, Role: FOLLOWER
I20250628 02:05:01.834841 1636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 63, Last appended: 10.63, Last appended by leader: 63, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:01.835417 1641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:05:01.835523 1641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:05:01.835660 1636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: configured and running, proceeding with master startup.
I20250628 02:05:01.839653 1566 master_runner.cc:186] Error getting master registration for 127.30.198.62:42671: Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111),
I20250628 02:05:01.841406 1566 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:05:01.841744 1655 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:05:01.841784 1655 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: loading cluster ID for follower catalog manager: success
I20250628 02:05:01.843497 1655 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: acquiring CA information for follower catalog manager: success
I20250628 02:05:01.844731 1655 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250628 02:05:01.967849 32300 heartbeater.cc:344] Connected to a master server at 127.30.198.60:40321
I20250628 02:05:01.969317 1600 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" instance_seqno: 1751076273975274) as {username='slave'} at 127.30.198.2:58213; Asking this server to re-register.
I20250628 02:05:01.969786 32300 heartbeater.cc:461] Registering TS with master...
I20250628 02:05:01.970896 1600 ts_manager.cc:194] Registered new tserver with Master: b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:05:02.171676 32165 heartbeater.cc:344] Connected to a master server at 127.30.198.60:40321
I20250628 02:05:02.172410 1600 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cb3d5d42c47546258deed617114e71f9" instance_seqno: 1751076273823804) as {username='slave'} at 127.30.198.1:52651; Asking this server to re-register.
I20250628 02:05:02.172736 32165 heartbeater.cc:461] Registering TS with master...
I20250628 02:05:02.173441 1600 ts_manager.cc:194] Registered new tserver with Master: cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:05:02.217551 1641 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:05:02.217725 1641 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:02.218806 1641 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
W20250628 02:05:02.219201 1586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:05:02.219938 1584 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:05:02.220036 1586 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:02.220093 1586 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 492448828d9a404fa245390eba0d56fb; no voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344
I20250628 02:05:02.220264 1641 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250628 02:05:02.231061 32435 heartbeater.cc:344] Connected to a master server at 127.30.198.60:40321
I20250628 02:05:02.231884 1599 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" instance_seqno: 1751076274129070) as {username='slave'} at 127.30.198.3:50559; Asking this server to re-register.
I20250628 02:05:02.232197 32435 heartbeater.cc:461] Registering TS with master...
I20250628 02:05:02.232873 1599 ts_manager.cc:194] Registered new tserver with Master: d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:05:02.775656 1669 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:05:02.775776 1669 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:02.776067 1669 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
W20250628 02:05:02.776962 1586 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:05:02.777047 1584 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
I20250628 02:05:02.777122 1584 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 492448828d9a404fa245390eba0d56fb; no voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344
I20250628 02:05:02.777236 1669 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250628 02:05:03.898454 1685 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:05:03.898559 1685 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:03.898912 1685 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
W20250628 02:05:03.899883 1584 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
W20250628 02:05:03.899959 1586 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:03.899995 1584 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 492448828d9a404fa245390eba0d56fb; no voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344
I20250628 02:05:03.900123 1685 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250628 02:05:04.777498 31512 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
/tmp/dist-test-taskXgtEGt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.62:42671
--webserver_interface=127.30.198.62
--webserver_port=40695
--builtin_ntp_servers=127.30.198.20:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250628 02:05:04.834210 1702 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:05:04.834378 1702 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:04.835039 1702 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
W20250628 02:05:04.835917 1586 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:05:04.835999 1584 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671): Network error: Client connection negotiation failed: client connection to 127.30.198.62:42671: connect: Connection refused (error 111)
I20250628 02:05:04.836058 1584 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 492448828d9a404fa245390eba0d56fb; no voters: 36911adab67246cfbfaf77ef5221fc92, e9fd188e41da4ec0b493c144accb0344
I20250628 02:05:04.836205 1702 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250628 02:05:04.867475 1701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 02:05:04.867671 1701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 02:05:04.867755 1701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 02:05:04.870460 1701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 02:05:04.870500 1701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 02:05:04.870517 1701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 02:05:04.870536 1701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 02:05:04.873579 1701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.198.20:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:36275
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.198.62:42671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.198.62
--webserver_port=40695
--never_fsync=true
--heap_profile_path=/tmp/kudu.1701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 01:43:12 UTC on 24a791456cd2
build id 6802
I20250628 02:05:04.874320 1701 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 02:05:04.875151 1701 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 02:05:04.880468 1710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:05:04.880506 1713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 02:05:04.880609 1711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 02:05:04.880609 1701 server_base.cc:1048] running on GCE node
I20250628 02:05:04.881011 1701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 02:05:04.881698 1701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 02:05:04.882828 1701 hybrid_clock.cc:648] HybridClock initialized: now 1751076304882808 us; error 31 us; skew 500 ppm
I20250628 02:05:04.888897 1701 webserver.cc:469] Webserver started at http://127.30.198.62:40695/ using document root <none> and password file <none>
I20250628 02:05:04.889323 1701 fs_manager.cc:362] Metadata directory not provided
I20250628 02:05:04.889370 1701 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 02:05:04.892009 1701 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20250628 02:05:04.893507 1719 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 02:05:04.894251 1701 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250628 02:05:04.894348 1701 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
uuid: "e9fd188e41da4ec0b493c144accb0344"
format_stamp: "Formatted at 2025-06-28 02:04:33 on dist-test-slave-0t45"
I20250628 02:05:04.894621 1701 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 02:05:04.904635 1701 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 02:05:04.905071 1701 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 02:05:04.905200 1701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 02:05:04.910806 1701 rpc_server.cc:307] RPC server started. Bound to: 127.30.198.62:42671
I20250628 02:05:04.910815 1771 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.198.62:42671 every 8 connection(s)
I20250628 02:05:04.911620 1701 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/master-0/data/info.pb
I20250628 02:05:04.912029 31512 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskXgtEGt/build/debug/bin/kudu as pid 1701
I20250628 02:05:04.915593 1772 sys_catalog.cc:263] Verifying existing consensus state
I20250628 02:05:04.917107 1772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Bootstrap starting.
I20250628 02:05:04.923794 1772 log.cc:826] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Log is configured to *not* fsync() on all Append() calls
I20250628 02:05:04.950700 1772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Bootstrap replayed 1/1 log segments. Stats: ops{read=59 overwritten=0 applied=59 ignored=0} inserts{seen=35 ignored=0} mutations{seen=41 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 02:05:04.950961 1772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: Bootstrap complete.
I20250628 02:05:04.953047 1772 raft_consensus.cc:357] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:04.953327 1772 raft_consensus.cc:738] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fd188e41da4ec0b493c144accb0344, State: Initialized, Role: FOLLOWER
I20250628 02:05:04.953689 1772 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 59, Last appended: 9.59, Last appended by leader: 59, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:04.953984 1776 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:05:04.954090 1776 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:05:04.954443 1772 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 02:05:04.959390 1790 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:05:04.959472 1790 catalog_manager.cc:1554] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: loading cluster ID for follower catalog manager: success
I20250628 02:05:04.961670 1790 catalog_manager.cc:1576] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: acquiring CA information for follower catalog manager: success
I20250628 02:05:04.963194 1790 catalog_manager.cc:1604] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250628 02:05:04.963387 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:05:04.965042 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:05:04.975360 32301 heartbeater.cc:344] Connected to a master server at 127.30.198.62:42671
I20250628 02:05:04.977082 1736 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" instance_seqno: 1751076273975274) as {username='slave'} at 127.30.198.2:59079; Asking this server to re-register.
I20250628 02:05:04.977571 32301 heartbeater.cc:461] Registering TS with master...
I20250628 02:05:04.978790 1736 ts_manager.cc:194] Registered new tserver with Master: b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:05:05.175060 32166 heartbeater.cc:344] Connected to a master server at 127.30.198.62:42671
I20250628 02:05:05.175899 1736 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cb3d5d42c47546258deed617114e71f9" instance_seqno: 1751076273823804) as {username='slave'} at 127.30.198.1:35787; Asking this server to re-register.
I20250628 02:05:05.176247 32166 heartbeater.cc:461] Registering TS with master...
I20250628 02:05:05.176932 1736 ts_manager.cc:194] Registered new tserver with Master: cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393)
I20250628 02:05:05.235121 32436 heartbeater.cc:344] Connected to a master server at 127.30.198.62:42671
I20250628 02:05:05.235982 1736 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" instance_seqno: 1751076274129070) as {username='slave'} at 127.30.198.3:58763; Asking this server to re-register.
I20250628 02:05:05.236383 32436 heartbeater.cc:461] Registering TS with master...
I20250628 02:05:05.237036 1736 ts_manager.cc:194] Registered new tserver with Master: d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733)
I20250628 02:05:05.557004 1801 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:05:05.557188 1801 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:05.558334 1801 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:05:05.558818 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:05:05.559522 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:05.561817 1610 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 10 candidate_status { last_received { term: 9 index: 59 } } ignore_live_leader: false dest_uuid: "492448828d9a404fa245390eba0d56fb" is_pre_election: true
I20250628 02:05:05.562067 1610 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e9fd188e41da4ec0b493c144accb0344 in current term 10: Already voted for candidate 36911adab67246cfbfaf77ef5221fc92 in this term.
I20250628 02:05:05.562420 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:05:05.562613 1801 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 9 FOLLOWER]: Advancing to term 10
I20250628 02:05:05.564198 1801 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250628 02:05:06.282850 1809 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:05:06.282949 1809 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:06.283353 1809 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
I20250628 02:05:06.283756 1610 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9fd188e41da4ec0b493c144accb0344" candidate_term: 11 candidate_status { last_received { term: 9 index: 59 } } ignore_live_leader: false dest_uuid: "492448828d9a404fa245390eba0d56fb" is_pre_election: true
I20250628 02:05:06.283924 1610 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e9fd188e41da4ec0b493c144accb0344 for term 11 because replica has last-logged OpId of term: 10 index: 63, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 59.
W20250628 02:05:06.284406 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:06.284461 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:05:06.284560 1809 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250628 02:05:06.846602 1566 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:05:06.846709 1566 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:05:06.850849 1818 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:05:06.850924 1818 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:06.851222 1818 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
I20250628 02:05:06.851534 1746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492448828d9a404fa245390eba0d56fb" candidate_term: 11 candidate_status { last_received { term: 10 index: 63 } } ignore_live_leader: false dest_uuid: "e9fd188e41da4ec0b493c144accb0344" is_pre_election: true
I20250628 02:05:06.851750 1746 raft_consensus.cc:2466] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448828d9a404fa245390eba0d56fb in term 10.
I20250628 02:05:06.852072 1584 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344; no voters:
W20250628 02:05:06.852178 1586 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:06.852228 1818 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250628 02:05:06.852285 1818 raft_consensus.cc:491] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:05:06.852329 1818 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 10 FOLLOWER]: Advancing to term 11
I20250628 02:05:06.853801 1818 raft_consensus.cc:513] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:06.854065 1818 leader_election.cc:290] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 election: Requested vote from peers e9fd188e41da4ec0b493c144accb0344 (127.30.198.62:42671), 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059)
I20250628 02:05:06.854311 1746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492448828d9a404fa245390eba0d56fb" candidate_term: 11 candidate_status { last_received { term: 10 index: 63 } } ignore_live_leader: false dest_uuid: "e9fd188e41da4ec0b493c144accb0344"
I20250628 02:05:06.854405 1746 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 10 FOLLOWER]: Advancing to term 11
W20250628 02:05:06.854846 1586 leader_election.cc:336] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:06.855293 1746 raft_consensus.cc:2466] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448828d9a404fa245390eba0d56fb in term 11.
I20250628 02:05:06.855523 1584 leader_election.cc:304] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 492448828d9a404fa245390eba0d56fb, e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92
I20250628 02:05:06.855664 1818 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 11 FOLLOWER]: Leader election won for term 11
I20250628 02:05:06.855830 1818 raft_consensus.cc:695] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [term 11 LEADER]: Becoming Leader. State: Replica: 492448828d9a404fa245390eba0d56fb, State: Running, Role: LEADER
I20250628 02:05:06.856153 1818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 10.63, Last appended by leader: 63, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:06.857436 1820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 492448828d9a404fa245390eba0d56fb. Latest consensus state: current_term: 11 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:05:06.857527 1820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: LEADER
I20250628 02:05:06.857785 1822 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 02:05:06.859290 1822 catalog_manager.cc:671] Loaded metadata for table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003]
I20250628 02:05:06.859544 1822 catalog_manager.cc:671] Loaded metadata for table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0]
I20250628 02:05:06.859781 1822 catalog_manager.cc:671] Loaded metadata for table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb]
I20250628 02:05:06.860019 1822 catalog_manager.cc:671] Loaded metadata for table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc]
I20250628 02:05:06.860219 1822 catalog_manager.cc:671] Loaded metadata for table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]
I20250628 02:05:06.860404 1822 catalog_manager.cc:671] Loaded metadata for table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c]
I20250628 02:05:06.860591 1822 catalog_manager.cc:671] Loaded metadata for table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e]
I20250628 02:05:06.861605 1822 tablet_loader.cc:96] loaded metadata for tablet 31216cf12a464b769be633b41088c417 (table default.table_d40b0217227a48fc9431fb713c4ed6be [id=c726262dad90499b8c9fda125fda1c5c])
I20250628 02:05:06.862111 1822 tablet_loader.cc:96] loaded metadata for tablet 70e6b67d5b87462a96838d8229f0e13a (table default.table_48852943c5c64ddfbde3e614d034c8d6 [id=17962a5794864b1583c98a76fe18b003])
I20250628 02:05:06.862262 1822 tablet_loader.cc:96] loaded metadata for tablet 73d2f50340b14bd08abd193108c40b47 (table default.table_25f7e6e53154401aa745623d0a9238e5 [id=9c530e20ae3f4e529c4d6936203fe1dc])
I20250628 02:05:06.862406 1822 tablet_loader.cc:96] loaded metadata for tablet 7d751e2d40b94c3ebf4faaa9e621fc20 (table default.table_51acb52242224cffa3ce3ea99517351a [id=582401feb808444b885f2532faac36e0])
I20250628 02:05:06.862938 1822 tablet_loader.cc:96] loaded metadata for tablet bfff6a29b6c34a1291ee7c92793829f7 (table default.table_aeae2af4c3b941689431955a4b4cacbd [id=72b2e754908e45da8d264fd73e392ffb])
I20250628 02:05:06.863081 1822 tablet_loader.cc:96] loaded metadata for tablet c8d15f43cc6c4adc85f5f1a532c9092b (table default.table_5f71bbbfb6c04933890499752142ceb3 [id=f244a9cbbe994d778333a82b209ffe5e])
I20250628 02:05:06.863442 1822 tablet_loader.cc:96] loaded metadata for tablet f1a4e82fb4204029b5675791641b0629 (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3])
I20250628 02:05:06.863530 1822 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 02:05:06.863974 1822 catalog_manager.cc:1261] Loaded cluster ID: cb064144d22c4ff381e14197e22757dc
I20250628 02:05:06.864014 1822 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 02:05:06.864557 1822 catalog_manager.cc:1506] Loading token signing keys...
I20250628 02:05:06.864979 1822 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Loaded TSK: 0
I20250628 02:05:06.865551 1822 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 02:05:06.865995 1822 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250628 02:05:06.866411 1822 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 19
I20250628 02:05:06.926890 1599 master_service.cc:948] ReplaceTablet: received request to replace tablet e6af82fc366e4121967e834148998420 from {username='slave'} at 127.0.0.1:36148
I20250628 02:05:06.926990 1600 master_service.cc:948] ReplaceTablet: received request to replace tablet f1a4e82fb4204029b5675791641b0629 from {username='slave'} at 127.0.0.1:36148
I20250628 02:05:06.927434 1595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36148:
name: "default.Table_0ba2136e171748f0ac70f94ad2378a15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:05:06.927521 1592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36148:
name: "default.Table_48b5e32835b14bc489b2a0c3737240a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:05:06.927632 1597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36148:
table { table_name: "default.Table_aeae2af4c3b941689431955a4b4cacbd" } modify_external_catalogs: true
I20250628 02:05:06.927927 1594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36148:
table { table_name: "default.Table_51acb52242224cffa3ce3ea99517351a" } modify_external_catalogs: true
I20250628 02:05:06.928010 1597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36148:
table { table_name: "default.Table_aeae2af4c3b941689431955a4b4cacbd" } modify_external_catalogs: true
I20250628 02:05:06.928043 1594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36148:
table { table_name: "default.Table_51acb52242224cffa3ce3ea99517351a" } modify_external_catalogs: true
I20250628 02:05:06.927537 1598 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36148:
table { table_name: "default.Table_5f71bbbfb6c04933890499752142ceb3" } new_table_name: "default.Table_4d41a145f10944dca41a0675451382fc" modify_external_catalogs: true
I20250628 02:05:06.927847 1596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36148:
name: "default.Table_f0ea02c8f8f944ebb5b8006e13d1f24c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:05:06.928251 1593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36148:
table { table_name: "default.Table_ccdc9aecb2d34778937160fc3d5cb4cc" } modify_external_catalogs: true
I20250628 02:05:06.928352 1593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36148:
table { table_name: "default.Table_ccdc9aecb2d34778937160fc3d5cb4cc" } modify_external_catalogs: true
I20250628 02:05:06.928369 1599 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36148:
name: "default.Table_57afb1ada3224fe1a269b48cdb5e87ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:05:06.928259 1591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36148:
name: "default.Table_6296f6b4fab44283a0eb5a6ee76c4306"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 02:05:06.930805 1746 raft_consensus.cc:1273] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Refusing update from remote peer 492448828d9a404fa245390eba0d56fb: Log matching property violated. Preceding OpId in replica: term: 9 index: 59. Preceding OpId from leader: term: 11 index: 65. (index mismatch)
W20250628 02:05:06.931222 1586 consensus_peers.cc:489] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb -> Peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Couldn't send request to peer 36911adab67246cfbfaf77ef5221fc92. Status: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250628 02:05:06.931277 1820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 59, Time since last communication: 0.000s
F20250628 02:05:06.934216 1654 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 5 out of 10 received events: NotificationEvent(eventId=20, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_ccdc9aecb2d34778937160fc3d5cb4cc, message={"server":"","servicePrincipal":"","db":"default","table":"table_ccdc9aecb2d34778937160fc3d5cb4cc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_ccdc9aecb2d34778937160fc3d5cb4cc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ccdc9aecb2d34778937160fc3d5cb4cc\",\"kudu.table_id\":\"ae7629d68a7d4fdd9ed00f4b1b046cd3\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_0ba2136e171748f0ac70f94ad2378a15, message={"server":"","servicePrincipal":"","db":"default","table":"table_0ba2136e171748f0ac70f94ad2378a15","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0ba2136e171748f0ac70f94ad2378a15\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_0ba2136e171748f0ac70f94ad2378a15\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0ba2136e171748f0ac70f94ad2378a15\",\"kudu.table_id\":\"56f325f0d2044b03bda5136a7f717212\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1751076275, eventType=ALTER_TABLE, dbName=default, tableName=table_2ffede105ab7483388016f792547fd7f, message={"server":"","servicePrincipal":"","db":"default","table":"table_25f7e6e53154401aa745623d0a9238e5","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_25f7e6e53154401aa745623d0a9238e5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_25f7e6e53154401aa745623d0a9238e5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_25f7e6e53154401aa745623d0a9238e5\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_2ffede105ab7483388016f792547fd7f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_2ffede105ab7483388016f792547fd7f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2ffede105ab7483388016f792547fd7f\",\"kudu.table_id\":\"9c530e20ae3f4e529c4d6936203fe1dc\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=DROP_TABLE, dbName=default, tableName=table_48b5e32835b14bc489b2a0c3737240a3, message={"server":"","servicePrincipal":"","db":"default","table":"table_48b5e32835b14bc489b2a0c3737240a3","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_48b5e32835b14bc489b2a0c3737240a3\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076274},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_48b5e32835b14bc489b2a0c3737240a3\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_48b5e32835b14bc489b2a0c3737240a3\",\"kudu.table_id\":\"92619a7eecd54094bd8c3e908c2314e3\",\"transient_lastDdlTime\":\"1751076274\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1751076275, eventType=CREATE_TABLE, dbName=default, tableName=table_51acb52242224cffa3ce3ea99517351a, message={"server":"","servicePrincipal":"","db":"default","table":"table_51acb52242224cffa3ce3ea99517351a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_51acb52242224cffa3ce3ea99517351a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1751076275},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/warehouse/table_51acb52242224cffa3ce3ea99517351a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_51acb52242224cffa3ce3ea99517351a\",\"kudu.table_id\":\"582401feb808444b885f2532faac36e0\",\"transient_lastDdlTime\":\"1751076275\",\"comment\":\"\",\"kudu.master_addresses\":\"127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cb064144d22c4ff381e14197e22757dc\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1751076275,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250628 02:05:06.935153 1827 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:05:06.935294 1827 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:05:06.937005 1820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 11 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:05:06.937105 1820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb [sys.catalog]: This master's current role is: LEADER
I20250628 02:05:06.937227 1823 mvcc.cc:204] Tried to move back new op lower bound from 7172408553184350208 to 7172408552884592640. Current Snapshot: MvccSnapshot[applied={T|T < 7172408553184350208}]
@ 0x7faec6d81dcd google::LogMessage::Fail() at ??:0
I20250628 02:05:06.937896 1825 mvcc.cc:204] Tried to move back new op lower bound from 7172408522109255680 to 7172408521895301120. Current Snapshot: MvccSnapshot[applied={T|T < 7172408522109255680}]
I20250628 02:05:06.938529 1827 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:05:06.938616 1827 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7faec6d85b93 google::LogMessage::SendToLog() at ??:0
I20250628 02:05:06.939277 1824 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 492448828d9a404fa245390eba0d56fb. Latest consensus state: current_term: 11 leader_uuid: "492448828d9a404fa245390eba0d56fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } } }
I20250628 02:05:06.939384 1824 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [sys.catalog]: This master's current role is: FOLLOWER
I20250628 02:05:06.939885 1600 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 492448828d9a404fa245390eba0d56fb: Sending DeleteTablet for 3 replicas of tablet f1a4e82fb4204029b5675791641b0629
@ 0x7faec6d817cc google::LogMessage::Flush() at ??:0
I20250628 02:05:06.940995 1600 catalog_manager.cc:6652] ReplaceTablet: tablet f1a4e82fb4204029b5675791641b0629 deleted and replaced by tablet 6dfce96d1d354f789445ca6b62b008f4
I20250628 02:05:06.941262 1600 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36148:
name: "default.Table_0f9f68e3f7dd46c29954d59183c6b66f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7faec6d81a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7faecbd1ac91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250628 02:05:06.946867 32233 tablet_service.cc:1515] Processing DeleteTablet for tablet f1a4e82fb4204029b5675791641b0629 with delete_type TABLET_DATA_DELETED (replaced by tablet 6dfce96d1d354f789445ca6b62b008f4) from {username='slave'} at 127.0.0.1:49470
I20250628 02:05:06.947062 32230 tablet_service.cc:1468] Processing CreateTablet for tablet 6dfce96d1d354f789445ca6b62b008f4 (DEFAULT_TABLE table=default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:05:06.947357 32230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dfce96d1d354f789445ca6b62b008f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:05:06.948120 32367 tablet_service.cc:1515] Processing DeleteTablet for tablet f1a4e82fb4204029b5675791641b0629 with delete_type TABLET_DATA_DELETED (replaced by tablet 6dfce96d1d354f789445ca6b62b008f4) from {username='slave'} at 127.0.0.1:55812
I20250628 02:05:06.948365 32368 tablet_service.cc:1468] Processing CreateTablet for tablet 6dfce96d1d354f789445ca6b62b008f4 (DEFAULT_TABLE table=default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:05:06.948611 32368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dfce96d1d354f789445ca6b62b008f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:05:06.950367 1841 tablet_bootstrap.cc:492] T 6dfce96d1d354f789445ca6b62b008f4 P b9085abee1ab4c96a3716383fe8fe592: Bootstrap starting.
I20250628 02:05:06.950542 1840 tablet_replica.cc:331] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685: stopping tablet replica
I20250628 02:05:06.950649 1840 raft_consensus.cc:2241] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:05:06.950766 1840 raft_consensus.cc:2270] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:05:06.950773 32098 tablet_service.cc:1515] Processing DeleteTablet for tablet f1a4e82fb4204029b5675791641b0629 with delete_type TABLET_DATA_DELETED (replaced by tablet 6dfce96d1d354f789445ca6b62b008f4) from {username='slave'} at 127.0.0.1:48768
I20250628 02:05:06.951149 1840 ts_tablet_manager.cc:1905] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:05:06.951165 32096 tablet_service.cc:1468] Processing CreateTablet for tablet 6dfce96d1d354f789445ca6b62b008f4 (DEFAULT_TABLE table=default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 02:05:06.951236 1841 tablet_bootstrap.cc:654] T 6dfce96d1d354f789445ca6b62b008f4 P b9085abee1ab4c96a3716383fe8fe592: Neither blocks nor log segments found. Creating new log.
I20250628 02:05:06.951407 32096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dfce96d1d354f789445ca6b62b008f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 02:05:06.951910 1843 tablet_bootstrap.cc:492] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685: Bootstrap starting.
I20250628 02:05:06.952821 1843 tablet_bootstrap.cc:654] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685: Neither blocks nor log segments found. Creating new log.
I20250628 02:05:06.953044 1840 ts_tablet_manager.cc:1918] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:05:06.953117 1840 log.cc:1199] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-2/wal/wals/f1a4e82fb4204029b5675791641b0629
I20250628 02:05:06.953365 1840 ts_tablet_manager.cc:1939] T f1a4e82fb4204029b5675791641b0629 P d0aa6633f9b6481bb29b00d602764685: Deleting consensus metadata
I20250628 02:05:06.953591 1844 tablet_replica.cc:331] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9: stopping tablet replica
I20250628 02:05:06.953693 1844 raft_consensus.cc:2241] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 02:05:06.953778 1844 raft_consensus.cc:2270] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:05:06.953928 1585 catalog_manager.cc:4928] TS d0aa6633f9b6481bb29b00d602764685 (127.30.198.3:34733): tablet f1a4e82fb4204029b5675791641b0629 (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]) successfully deleted
@ 0x7faecbd19ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250628 02:05:06.954108 1844 ts_tablet_manager.cc:1905] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:05:06.954190 1839 tablet_replica.cc:331] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592: stopping tablet replica
I20250628 02:05:06.954319 1839 raft_consensus.cc:2241] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [term 1 LEADER]: Raft consensus shutting down.
I20250628 02:05:06.954497 1839 raft_consensus.cc:2270] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 02:05:06.954842 1839 ts_tablet_manager.cc:1905] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 02:05:06.955488 1841 tablet_bootstrap.cc:492] T 6dfce96d1d354f789445ca6b62b008f4 P b9085abee1ab4c96a3716383fe8fe592: No bootstrap required, opened a new log
I20250628 02:05:06.955559 1841 ts_tablet_manager.cc:1397] T 6dfce96d1d354f789445ca6b62b008f4 P b9085abee1ab4c96a3716383fe8fe592: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250628 02:05:06.956058 1841 raft_consensus.cc:357] T 6dfce96d1d354f789445ca6b62b008f4 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:05:06.956216 1841 raft_consensus.cc:383] T 6dfce96d1d354f789445ca6b62b008f4 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:05:06.956249 1841 raft_consensus.cc:738] T 6dfce96d1d354f789445ca6b62b008f4 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9085abee1ab4c96a3716383fe8fe592, State: Initialized, Role: FOLLOWER
I20250628 02:05:06.956449 1841 consensus_queue.cc:260] T 6dfce96d1d354f789445ca6b62b008f4 P b9085abee1ab4c96a3716383fe8fe592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
@ 0x7faecbd1930b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250628 02:05:06.957141 1839 ts_tablet_manager.cc:1918] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:05:06.957203 1839 log.cc:1199] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-1/wal/wals/f1a4e82fb4204029b5675791641b0629
I20250628 02:05:06.957442 1839 ts_tablet_manager.cc:1939] T f1a4e82fb4204029b5675791641b0629 P b9085abee1ab4c96a3716383fe8fe592: Deleting consensus metadata
I20250628 02:05:06.957471 1847 tablet_bootstrap.cc:492] T 6dfce96d1d354f789445ca6b62b008f4 P cb3d5d42c47546258deed617114e71f9: Bootstrap starting.
I20250628 02:05:06.958253 1841 ts_tablet_manager.cc:1428] T 6dfce96d1d354f789445ca6b62b008f4 P b9085abee1ab4c96a3716383fe8fe592: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250628 02:05:06.958424 1847 tablet_bootstrap.cc:654] T 6dfce96d1d354f789445ca6b62b008f4 P cb3d5d42c47546258deed617114e71f9: Neither blocks nor log segments found. Creating new log.
I20250628 02:05:06.959088 1587 catalog_manager.cc:4928] TS b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697): tablet f1a4e82fb4204029b5675791641b0629 (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]) successfully deleted
@ 0x7faecbd1ea12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x555795d93752 std::function<>::operator()() at ??:0
I20250628 02:05:06.960216 1844 ts_tablet_manager.cc:1918] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250628 02:05:06.960281 1844 log.cc:1199] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9: Deleting WAL directory at /tmp/dist-test-taskXgtEGt/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1751076260811550-31512-0/minicluster-data/ts-0/wal/wals/f1a4e82fb4204029b5675791641b0629
I20250628 02:05:06.960518 1844 ts_tablet_manager.cc:1939] T f1a4e82fb4204029b5675791641b0629 P cb3d5d42c47546258deed617114e71f9: Deleting consensus metadata
I20250628 02:05:06.961123 1584 catalog_manager.cc:4928] TS cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393): tablet f1a4e82fb4204029b5675791641b0629 (table default.table_ccdc9aecb2d34778937160fc3d5cb4cc [id=ae7629d68a7d4fdd9ed00f4b1b046cd3]) successfully deleted
I20250628 02:05:06.961586 1847 tablet_bootstrap.cc:492] T 6dfce96d1d354f789445ca6b62b008f4 P cb3d5d42c47546258deed617114e71f9: No bootstrap required, opened a new log
I20250628 02:05:06.961651 1847 ts_tablet_manager.cc:1397] T 6dfce96d1d354f789445ca6b62b008f4 P cb3d5d42c47546258deed617114e71f9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250628 02:05:06.961793 1843 tablet_bootstrap.cc:492] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685: No bootstrap required, opened a new log
I20250628 02:05:06.961862 1843 ts_tablet_manager.cc:1397] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250628 02:05:06.962133 1847 raft_consensus.cc:357] T 6dfce96d1d354f789445ca6b62b008f4 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:05:06.962217 1847 raft_consensus.cc:383] T 6dfce96d1d354f789445ca6b62b008f4 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
@ 0x7faec81d30d1 kudu::Thread::SuperviseThread() at ??:0
I20250628 02:05:06.962246 1847 raft_consensus.cc:738] T 6dfce96d1d354f789445ca6b62b008f4 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3d5d42c47546258deed617114e71f9, State: Initialized, Role: FOLLOWER
I20250628 02:05:06.962213 1843 raft_consensus.cc:357] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:05:06.962291 1843 raft_consensus.cc:383] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 02:05:06.962329 1843 raft_consensus.cc:738] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Initialized, Role: FOLLOWER
I20250628 02:05:06.962356 1847 consensus_queue.cc:260] T 6dfce96d1d354f789445ca6b62b008f4 P cb3d5d42c47546258deed617114e71f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:05:06.962448 1843 consensus_queue.cc:260] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:05:06.962662 1847 ts_tablet_manager.cc:1428] T 6dfce96d1d354f789445ca6b62b008f4 P cb3d5d42c47546258deed617114e71f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
@ 0x7faec89c26db start_thread at ??:0
I20250628 02:05:06.965029 1843 ts_tablet_manager.cc:1428] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
@ 0x7faec5f3871f clone at ??:0
I20250628 02:05:06.983510 31512 master-stress-test.cc:577] Killing test thread 1/25
W20250628 02:05:07.025249 32303 tablet.cc:2378] T 6dfce96d1d354f789445ca6b62b008f4 P b9085abee1ab4c96a3716383fe8fe592: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 02:05:07.099033 32443 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.30.198.60:40321)
W20250628 02:05:07.099267 32443 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.30.198.60:40321)
W20250628 02:05:07.099401 32443 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.198.60:40321)
W20250628 02:05:07.099561 32443 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.30.198.60:40321)
I20250628 02:05:07.109017 1815 raft_consensus.cc:491] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 02:05:07.109113 1815 raft_consensus.cc:513] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:05:07.109493 1815 leader_election.cc:290] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:05:07.109751 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dfce96d1d354f789445ca6b62b008f4" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9" is_pre_election: true
I20250628 02:05:07.109790 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dfce96d1d354f789445ca6b62b008f4" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592" is_pre_election: true
I20250628 02:05:07.109867 32119 raft_consensus.cc:2466] T 6dfce96d1d354f789445ca6b62b008f4 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 0.
I20250628 02:05:07.109925 32254 raft_consensus.cc:2466] T 6dfce96d1d354f789445ca6b62b008f4 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 0.
I20250628 02:05:07.110271 32325 leader_election.cc:304] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:05:07.110531 1815 raft_consensus.cc:2802] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 02:05:07.110581 1815 raft_consensus.cc:491] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 02:05:07.110620 1815 raft_consensus.cc:3058] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:05:07.111457 1815 raft_consensus.cc:513] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
I20250628 02:05:07.111819 1815 leader_election.cc:290] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 election: Requested vote from peers cb3d5d42c47546258deed617114e71f9 (127.30.198.1:37393), b9085abee1ab4c96a3716383fe8fe592 (127.30.198.2:46697)
I20250628 02:05:07.112120 32254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dfce96d1d354f789445ca6b62b008f4" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9085abee1ab4c96a3716383fe8fe592"
I20250628 02:05:07.112149 32119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dfce96d1d354f789445ca6b62b008f4" candidate_uuid: "d0aa6633f9b6481bb29b00d602764685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3d5d42c47546258deed617114e71f9"
I20250628 02:05:07.112219 32254 raft_consensus.cc:3058] T 6dfce96d1d354f789445ca6b62b008f4 P b9085abee1ab4c96a3716383fe8fe592 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:05:07.112244 32119 raft_consensus.cc:3058] T 6dfce96d1d354f789445ca6b62b008f4 P cb3d5d42c47546258deed617114e71f9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 02:05:07.112870 32254 raft_consensus.cc:2466] T 6dfce96d1d354f789445ca6b62b008f4 P b9085abee1ab4c96a3716383fe8fe592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 1.
I20250628 02:05:07.112881 32119 raft_consensus.cc:2466] T 6dfce96d1d354f789445ca6b62b008f4 P cb3d5d42c47546258deed617114e71f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0aa6633f9b6481bb29b00d602764685 in term 1.
I20250628 02:05:07.113121 32325 leader_election.cc:304] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9085abee1ab4c96a3716383fe8fe592, d0aa6633f9b6481bb29b00d602764685; no voters:
I20250628 02:05:07.113262 1815 raft_consensus.cc:2802] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 02:05:07.113327 1815 raft_consensus.cc:695] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [term 1 LEADER]: Becoming Leader. State: Replica: d0aa6633f9b6481bb29b00d602764685, State: Running, Role: LEADER
I20250628 02:05:07.113448 1815 consensus_queue.cc:237] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 } } peers { permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 } } peers { permanent_uuid: "d0aa6633f9b6481bb29b00d602764685" member_type: VOTER last_known_addr { host: "127.30.198.3" port: 34733 } }
W20250628 02:05:07.127379 32168 tablet.cc:2378] T 6dfce96d1d354f789445ca6b62b008f4 P cb3d5d42c47546258deed617114e71f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 02:05:07.181882 32438 tablet.cc:2378] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 02:05:07.448100 1863 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:05:07.448212 1863 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:07.448590 1863 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:05:07.449837 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:05:07.449908 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:05:07.449956 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:05:07.450091 1863 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:05:07.519320 1815 consensus_queue.cc:1035] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9085abee1ab4c96a3716383fe8fe592" member_type: VOTER last_known_addr { host: "127.30.198.2" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:05:07.522367 1557 consensus_queue.cc:1035] T 6dfce96d1d354f789445ca6b62b008f4 P d0aa6633f9b6481bb29b00d602764685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3d5d42c47546258deed617114e71f9" member_type: VOTER last_known_addr { host: "127.30.198.1" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 02:05:08.137630 1879 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:05:08.137730 1879 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:08.138120 1879 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:05:08.139276 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:05:08.139360 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:08.139434 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:05:08.139541 1879 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:05:08.829146 1888 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:05:08.829272 1888 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:08.829609 1888 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:05:08.830628 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:05:08.830698 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:05:08.830744 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:05:08.830855 1888 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:05:09.966427 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:05:09.966516 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:05:10.516567 1909 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:05:10.516696 1909 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:10.517108 1909 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:05:10.518049 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:05:10.518143 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:10.518203 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:05:10.518333 1909 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:05:11.232935 1923 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:05:11.233040 1923 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:11.233373 1923 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:05:11.233788 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250628 02:05:11.234436 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:05:11.234516 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:11.234588 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:05:11.234712 1923 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:05:14.582087 1968 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:05:14.582186 1968 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:14.582491 1968 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:05:14.583416 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:05:14.583617 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:14.583698 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:05:14.583827 1968 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:05:14.967865 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:05:14.967945 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:05:15.436944 1978 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:05:15.437057 1978 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:15.437397 1978 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:05:15.438293 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:05:15.438378 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:15.438462 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:05:15.438616 1978 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:05:19.971351 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:05:19.971451 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:05:20.748502 2043 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:05:20.748601 2043 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:20.748891 2043 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:05:20.749334 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250628 02:05:20.749997 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:05:20.750079 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:20.750140 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:05:20.750267 2043 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:05:23.150564 2077 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:05:23.150683 2077 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:23.151082 2077 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:05:23.152140 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:05:23.152215 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:05:23.152268 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:05:23.152387 2077 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:05:24.973315 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:05:24.973435 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:05:29.974853 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:05:29.974951 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:05:32.520620 2190 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:05:32.520742 2190 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:32.521121 2190 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:05:32.521623 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250628 02:05:32.522188 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:05:32.522259 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:05:32.522310 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:05:32.522419 2190 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250628 02:05:34.567984 32300 heartbeater.cc:646] Failed to heartbeat to 127.30.198.60:40321 (57 consecutive failures): Network error: Failed to ping master at 127.30.198.60:40321: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:05:34.568073 32167 heartbeater.cc:646] Failed to heartbeat to 127.30.198.61:35059 (74 consecutive failures): Network error: Failed to ping master at 127.30.198.61:35059: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:34.976563 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:05:34.976655 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250628 02:05:35.042239 32437 heartbeater.cc:646] Failed to heartbeat to 127.30.198.61:35059 (80 consecutive failures): Network error: Failed to ping master at 127.30.198.61:35059: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:39.978197 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:05:39.978291 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:05:44.979840 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:05:44.979928 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:05:49.981449 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:05:49.981534 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:05:52.366695 2430 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:05:52.366801 2430 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:05:52.367203 2430 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:05:52.367745 1721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:05:52.368422 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:05:52.368502 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:05:52.368592 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:05:52.368709 2430 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:05:54.983245 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:05:54.983331 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:05:59.984951 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:05:59.985038 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:06:01.882308 2546 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:06:01.882412 2546 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:06:01.882756 2546 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:06:01.883195 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:06:01.883891 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:06:01.883970 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:06:01.884037 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:06:01.884167 2546 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:06:04.006767 2574 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:06:04.006886 2574 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:06:04.007293 2574 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:06:04.008494 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:06:04.008648 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:06:04.008711 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:06:04.008859 2574 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:06:04.986838 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:06:04.986924 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:06:09.988641 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:06:09.988745 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:06:14.990288 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:06:14.990383 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:06:19.364302 2759 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:06:19.364414 2759 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:06:19.364804 2759 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:06:19.365330 1721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250628 02:06:19.366045 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:06:19.366117 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:06:19.366169 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:06:19.366276 2759 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:06:19.992064 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:06:19.992163 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:06:24.993737 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:06:24.993867 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:06:29.995858 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:06:29.995990 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250628 02:06:34.669041 32300 heartbeater.cc:646] Failed to heartbeat to 127.30.198.60:40321 (177 consecutive failures): Network error: Failed to ping master at 127.30.198.60:40321: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:06:34.669230 32165 heartbeater.cc:646] Failed to heartbeat to 127.30.198.60:40321 (176 consecutive failures): Network error: Failed to ping master at 127.30.198.60:40321: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:06:34.997725 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:06:34.997824 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250628 02:06:35.139453 32437 heartbeater.cc:646] Failed to heartbeat to 127.30.198.61:35059 (200 consecutive failures): Network error: Failed to ping master at 127.30.198.61:35059: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:06:35.139453 32435 heartbeater.cc:646] Failed to heartbeat to 127.30.198.60:40321 (179 consecutive failures): Network error: Failed to ping master at 127.30.198.60:40321: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:06:36.163652 2963 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:06:36.163780 2963 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:06:36.164147 2963 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:06:36.164547 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:06:36.165127 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:06:36.165208 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:06:36.165287 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:06:36.165400 2963 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:06:39.999531 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:06:39.999640 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:06:44.039574 3054 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:06:44.039721 3054 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:06:44.040174 3054 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:06:44.040614 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:06:44.041249 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:06:44.041313 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:06:44.041369 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:06:44.041478 3054 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:06:45.001101 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:06:45.001190 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:06:50.002701 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:06:50.002789 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:06:55.004288 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:06:55.004376 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:07:00.005899 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:07:00.005990 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:07:02.196197 3263 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:07:02.196300 3263 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:07:02.196717 3263 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:07:02.197117 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:07:02.197587 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:07:02.197680 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:07:02.197742 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:07:02.197837 3263 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:07:05.007591 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:07:05.007675 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:07:05.027345 1725 maintenance_manager.cc:419] P e9fd188e41da4ec0b493c144accb0344: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033354
I20250628 02:07:05.057734 1724 maintenance_manager.cc:643] P e9fd188e41da4ec0b493c144accb0344: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.030s user 0.014s sys 0.000s Metrics: {"bytes_written":16225,"cfile_init":1,"dirs.queue_time_us":271,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":758,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":374,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":35,"thread_start_us":129,"threads_started":2,"wal-append.queue_time_us":113}
I20250628 02:07:05.058391 1725 maintenance_manager.cc:419] P e9fd188e41da4ec0b493c144accb0344: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 2089 bytes on disk
I20250628 02:07:05.058821 1724 maintenance_manager.cc:643] P e9fd188e41da4ec0b493c144accb0344: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4}
I20250628 02:07:10.009135 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:07:10.009218 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:07:13.623332 3408 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:07:13.623456 3408 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:07:13.623910 3408 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:07:13.624362 1721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:07:13.625156 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:07:13.625236 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:07:13.625330 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:07:13.625465 3408 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:07:15.010984 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:07:15.011097 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:07:20.012562 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:07:20.012655 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:07:25.015981 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:07:25.016067 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:07:30.017531 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:07:30.017629 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:07:30.079653 3671 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:07:30.079754 3671 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:07:30.080164 3671 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:07:30.080488 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:07:30.081103 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:07:30.081189 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:07:30.081240 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:07:30.081367 3671 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250628 02:07:34.773995 32300 heartbeater.cc:646] Failed to heartbeat to 127.30.198.60:40321 (297 consecutive failures): Network error: Failed to ping master at 127.30.198.60:40321: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:07:34.774096 32165 heartbeater.cc:646] Failed to heartbeat to 127.30.198.60:40321 (296 consecutive failures): Network error: Failed to ping master at 127.30.198.60:40321: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:07:35.019121 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:07:35.019212 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250628 02:07:35.233845 32435 heartbeater.cc:646] Failed to heartbeat to 127.30.198.60:40321 (299 consecutive failures): Network error: Failed to ping master at 127.30.198.60:40321: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:07:40.020879 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:07:40.020967 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:07:45.022696 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:07:45.022795 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:07:46.308706 3930 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:07:46.308813 3930 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:07:46.309304 3930 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:07:46.309693 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:07:46.310362 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:07:46.310614 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:07:46.310681 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:07:46.310789 3930 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:07:50.024502 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:07:50.024607 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:07:55.026096 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:07:55.026193 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:08:00.027657 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:08:00.027781 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:08:00.232177 4137 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:08:00.232285 4137 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:08:00.232626 4137 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:08:00.232985 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:08:00.233640 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:08:00.233711 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:08:00.233762 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:08:00.233875 4137 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:08:01.878140 4168 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:08:01.878252 4168 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:08:01.878638 4168 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:08:01.879840 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:08:01.879966 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:08:01.880017 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:08:01.880177 4168 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:08:05.029225 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:08:05.029310 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:08:08.090961 4260 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:08:08.091065 4260 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:08:08.091470 4260 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:08:08.091985 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250628 02:08:08.091985 1721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:08:08.092639 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:08:08.092795 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:08:08.092840 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:08:08.092939 4260 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:08:10.030769 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:08:10.030867 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:08:15.032414 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:08:15.032509 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:08:20.033965 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:08:20.034073 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:08:24.004014 4501 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:08:24.004109 4501 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:08:24.004586 4501 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:08:24.005115 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:08:24.005779 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:08:24.005859 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:08:24.005928 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:08:24.006064 4501 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:08:25.035583 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:08:25.035672 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:08:30.037170 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:08:30.037253 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:08:32.937948 4642 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:08:32.938059 4642 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:08:32.938448 4642 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:08:32.938872 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:08:32.939401 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:08:32.939505 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:08:32.939554 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:08:32.939677 4642 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250628 02:08:34.880883 32165 heartbeater.cc:646] Failed to heartbeat to 127.30.198.60:40321 (416 consecutive failures): Network error: Failed to ping master at 127.30.198.60:40321: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:08:34.880961 32302 heartbeater.cc:646] Failed to heartbeat to 127.30.198.61:35059 (436 consecutive failures): Network error: Failed to ping master at 127.30.198.61:35059: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:08:35.038720 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:08:35.038818 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250628 02:08:35.333092 32437 heartbeater.cc:646] Failed to heartbeat to 127.30.198.61:35059 (440 consecutive failures): Network error: Failed to ping master at 127.30.198.61:35059: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:08:40.040300 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:08:40.040401 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:08:41.207248 4765 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:08:41.207347 4765 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:08:41.207643 4765 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:08:41.208133 1721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:08:41.208698 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:08:41.208794 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:08:41.208889 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:08:41.209007 4765 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:08:45.042038 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:08:45.042151 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:08:48.535365 4880 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:08:48.535475 4880 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:08:48.535821 4880 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:08:48.536232 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:08:48.536796 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:08:48.536927 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:08:48.537009 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:08:48.537139 4880 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:08:50.043859 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:08:50.043959 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:08:55.045528 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:08:55.045620 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:09:00.047346 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:09:00.047448 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:09:02.483274 5088 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:09:02.483381 5088 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:09:02.483762 5088 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:09:02.484128 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:09:02.484711 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:09:02.484786 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:09:02.484853 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:09:02.484966 5088 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:09:05.049180 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:09:05.049266 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:09:08.401192 5175 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:09:08.401312 5175 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:09:08.401695 5175 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:09:08.402154 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:09:08.402748 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:09:08.402871 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:09:08.402916 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:09:08.403038 5175 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:09:10.050704 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:09:10.050801 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:09:15.052625 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:09:15.052736 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:09:20.054205 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:09:20.054293 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:09:22.376502 5382 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:09:22.376623 5382 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:09:22.376998 5382 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:09:22.377512 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:09:22.378100 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:09:22.378175 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:09:22.378227 1722 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:09:22.378334 5382 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:09:25.055862 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:09:25.055963 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:09:26.676292 5454 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:09:26.676414 5454 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:09:26.676815 5454 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:09:26.677824 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:09:26.677919 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:09:26.677981 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:09:26.678105 5454 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:09:29.135739 5494 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:09:29.135857 5494 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:09:29.136166 5494 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:09:29.136708 1721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250628 02:09:29.137318 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:09:29.137411 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:09:29.137488 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:09:29.137616 5494 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:09:30.057530 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:09:30.057622 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250628 02:09:34.985139 32302 heartbeater.cc:646] Failed to heartbeat to 127.30.198.61:35059 (556 consecutive failures): Network error: Failed to ping master at 127.30.198.61:35059: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:09:34.985237 32165 heartbeater.cc:646] Failed to heartbeat to 127.30.198.60:40321 (536 consecutive failures): Network error: Failed to ping master at 127.30.198.60:40321: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:09:35.059266 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:09:35.059360 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250628 02:09:35.435506 32437 heartbeater.cc:646] Failed to heartbeat to 127.30.198.61:35059 (560 consecutive failures): Network error: Failed to ping master at 127.30.198.61:35059: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
I20250628 02:09:40.060953 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:09:40.061043 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:09:45.062592 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:09:45.062717 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:09:45.726934 5736 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:09:45.727034 5736 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:09:45.727444 5736 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:09:45.727893 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250628 02:09:45.728549 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:09:45.728637 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:09:45.728708 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:09:45.728849 5736 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:09:47.631786 5768 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 492448828d9a404fa245390eba0d56fb)
I20250628 02:09:47.631908 5768 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd188e41da4ec0b493c144accb0344" member_type: VOTER last_known_addr { host: "127.30.198.62" port: 42671 } } peers { permanent_uuid: "36911adab67246cfbfaf77ef5221fc92" member_type: VOTER last_known_addr { host: "127.30.198.61" port: 35059 } } peers { permanent_uuid: "492448828d9a404fa245390eba0d56fb" member_type: VOTER last_known_addr { host: "127.30.198.60" port: 40321 } }
I20250628 02:09:47.632273 5768 leader_election.cc:290] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059), 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321)
W20250628 02:09:47.633201 1722 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 36911adab67246cfbfaf77ef5221fc92 (127.30.198.61:35059): Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111)
W20250628 02:09:47.633280 1721 leader_election.cc:336] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 492448828d9a404fa245390eba0d56fb (127.30.198.60:40321): Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
I20250628 02:09:47.633356 1721 leader_election.cc:304] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9fd188e41da4ec0b493c144accb0344; no voters: 36911adab67246cfbfaf77ef5221fc92, 492448828d9a404fa245390eba0d56fb
I20250628 02:09:47.633466 5768 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e9fd188e41da4ec0b493c144accb0344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250628 02:09:50.064270 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:09:50.064364 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250628 02:09:55.065969 1701 master_runner.cc:186] Error getting master registration for 127.30.198.61:35059: Network error: Client connection negotiation failed: client connection to 127.30.198.61:35059: connect: Connection refused (error 111),
I20250628 02:09:55.066084 1701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250628 02:09:59.712738 32442 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:09:59.712910 32442 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:09:59.712945 32442 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
W20250628 02:09:59.712980 32442 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
F20250628 02:09:59.723372 32466 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1751076599 (unix time) try "date -d @1751076599" if you are using GNU date ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726799 32460 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0f9f68e3f7dd46c29954d59183c6b66f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726799 32460 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0f9f68e3f7dd46c29954d59183c6b66f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726861 32461 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba2136e171748f0ac70f94ad2378a15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726799 32460 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0f9f68e3f7dd46c29954d59183c6b66f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726861 32461 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba2136e171748f0ac70f94ad2378a15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726919 32475 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726799 32460 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0f9f68e3f7dd46c29954d59183c6b66f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726861 32461 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba2136e171748f0ac70f94ad2378a15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726919 32475 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726866 32462 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_48b5e32835b14bc489b2a0c3737240a3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800007b18) received by PID 31512 (TID 0x7fbfe3556700) from PID 31512; stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726799 32460 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0f9f68e3f7dd46c29954d59183c6b66f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726861 32461 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba2136e171748f0ac70f94ad2378a15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726919 32475 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726866 32462 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_48b5e32835b14bc489b2a0c3737240a3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.727986 32457 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6296f6b4fab44283a0eb5a6ee76c4306 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726799 32460 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0f9f68e3f7dd46c29954d59183c6b66f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726861 32461 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba2136e171748f0ac70f94ad2378a15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726919 32475 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726866 32462 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_48b5e32835b14bc489b2a0c3737240a3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.727986 32457 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6296f6b4fab44283a0eb5a6ee76c4306 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.728029 32455 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4687199de5174487b23a117d7fe19211 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726799 32460 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0f9f68e3f7dd46c29954d59183c6b66f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726861 32461 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba2136e171748f0ac70f94ad2378a15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726919 32475 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726866 32462 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_48b5e32835b14bc489b2a0c3737240a3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.727986 32457 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6296f6b4fab44283a0eb5a6ee76c4306 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.728029 32455 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4687199de5174487b23a117d7fe19211 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729125 32473 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726799 32460 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0f9f68e3f7dd46c29954d59183c6b66f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726861 32461 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba2136e171748f0ac70f94ad2378a15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726919 32475 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726866 32462 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_48b5e32835b14bc489b2a0c3737240a3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.727986 32457 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6296f6b4fab44283a0eb5a6ee76c4306 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.728029 32455 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4687199de5174487b23a117d7fe19211 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729125 32473 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729158 32474 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726799 32460 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0f9f68e3f7dd46c29954d59183c6b66f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726861 32461 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba2136e171748f0ac70f94ad2378a15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726919 32475 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726866 32462 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_48b5e32835b14bc489b2a0c3737240a3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.727986 32457 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6296f6b4fab44283a0eb5a6ee76c4306 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.728029 32455 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4687199de5174487b23a117d7fe19211 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729125 32473 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729158 32474 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729198 32456 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f0ea02c8f8f944ebb5b8006e13d1f24c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726799 32460 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0f9f68e3f7dd46c29954d59183c6b66f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726861 32461 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba2136e171748f0ac70f94ad2378a15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726919 32475 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726866 32462 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_48b5e32835b14bc489b2a0c3737240a3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.727986 32457 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6296f6b4fab44283a0eb5a6ee76c4306 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.728029 32455 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4687199de5174487b23a117d7fe19211 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729125 32473 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729158 32474 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729198 32456 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f0ea02c8f8f944ebb5b8006e13d1f24c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729298 32454 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51147aa1de1d4fac95229bd0283919f3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7fbffd228980 (unknown) at ??:0
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726799 32460 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0f9f68e3f7dd46c29954d59183c6b66f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726861 32461 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba2136e171748f0ac70f94ad2378a15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726919 32475 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726866 32462 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_48b5e32835b14bc489b2a0c3737240a3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.727986 32457 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6296f6b4fab44283a0eb5a6ee76c4306 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.728029 32455 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4687199de5174487b23a117d7fe19211 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729125 32473 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729158 32474 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729198 32456 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f0ea02c8f8f944ebb5b8006e13d1f24c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729298 32454 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51147aa1de1d4fac95229bd0283919f3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.730391 32476 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726799 32460 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0f9f68e3f7dd46c29954d59183c6b66f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726861 32461 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba2136e171748f0ac70f94ad2378a15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726919 32475 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726866 32462 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_48b5e32835b14bc489b2a0c3737240a3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.727986 32457 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6296f6b4fab44283a0eb5a6ee76c4306 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.728029 32455 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4687199de5174487b23a117d7fe19211 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729125 32473 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729158 32474 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729198 32456 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f0ea02c8f8f944ebb5b8006e13d1f24c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729298 32454 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51147aa1de1d4fac95229bd0283919f3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.730391 32476 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.730394 32470 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7fbffae3afb7 gsignal at ??:0
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726799 32460 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0f9f68e3f7dd46c29954d59183c6b66f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726861 32461 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba2136e171748f0ac70f94ad2378a15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726919 32475 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726866 32462 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_48b5e32835b14bc489b2a0c3737240a3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.727986 32457 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6296f6b4fab44283a0eb5a6ee76c4306 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.728029 32455 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4687199de5174487b23a117d7fe19211 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729125 32473 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729158 32474 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729198 32456 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f0ea02c8f8f944ebb5b8006e13d1f24c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729298 32454 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51147aa1de1d4fac95229bd0283919f3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.730391 32476 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.730394 32470 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.731611 32458 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7eb9297b9cef4c8e82bbdb793ba152cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726799 32460 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0f9f68e3f7dd46c29954d59183c6b66f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726861 32461 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba2136e171748f0ac70f94ad2378a15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726919 32475 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726866 32462 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_48b5e32835b14bc489b2a0c3737240a3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.727986 32457 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6296f6b4fab44283a0eb5a6ee76c4306 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.728029 32455 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4687199de5174487b23a117d7fe19211 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729125 32473 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729158 32474 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729198 32456 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f0ea02c8f8f944ebb5b8006e13d1f24c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729298 32454 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51147aa1de1d4fac95229bd0283919f3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.730391 32476 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.730394 32470 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.731611 32458 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7eb9297b9cef4c8e82bbdb793ba152cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.732719 32471 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250628 02:09:59.723379 32463 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bed4f3bffa6d40138ec95e58cc15fba0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.724500 32469 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725615 32467 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.725636 32453 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_57afb1ada3224fe1a269b48cdb5e87ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726799 32460 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0f9f68e3f7dd46c29954d59183c6b66f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726861 32461 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ba2136e171748f0ac70f94ad2378a15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726919 32475 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.726866 32462 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_48b5e32835b14bc489b2a0c3737240a3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.727986 32457 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6296f6b4fab44283a0eb5a6ee76c4306 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.728029 32455 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4687199de5174487b23a117d7fe19211 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729125 32473 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729158 32474 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729198 32456 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f0ea02c8f8f944ebb5b8006e13d1f24c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.729298 32454 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51147aa1de1d4fac95229bd0283919f3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.730391 32476 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.730394 32470 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.731611 32458 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7eb9297b9cef4c8e82bbdb793ba152cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.732719 32471 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)F20250628 02:09:59.732738 32472 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.30.198.62:42671,127.30.198.61:35059,127.30.198.60:40321, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.30.198.60:40321: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7fbffae3c921 abort at ??:0
@ 0x7fbffbca3dcd google::LogMessage::Fail() at ??:0
@ 0x7fbffbca7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fbffbca37cc google::LogMessage::Flush() at ??:0
@ 0x7fbffbca4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x560499b740d3 kudu::MasterStressTest::AlterTableThread() at ??:0
@ 0x560499b867b0 std::__invoke_impl<>() at ??:0
@ 0x560499b844ec std::__invoke<>() at ??:0
@ 0x560499b8bb3d _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
@ 0x560499b8ab41 std::thread::_Invoker<>::operator()() at ??:0
@ 0x560499b898be std::thread::_State_impl<>::_M_run() at ??:0
@ 0x7fbffb4c26df (unknown) at ??:0
@ 0x7fbffd21d6db start_thread at ??:0
@ 0x7fbffaf1d71f clone at ??:0