Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250901 01:52:58.210505 416 test_util.cc:276] Using random seed: -1585461099
2025-09-01T01:52:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T01:52:58Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task1fHXBS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task1fHXBS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/security.properties
2025-09-01 01:53:04: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task1fHXBS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task1fHXBS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250901 01:53:13.083338 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1fHXBS/build/asan/bin/kudu
/tmp/dist-test-task1fHXBS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35805
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250901 01:53:13.469781 658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:53:13.470188 658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:53:13.470330 658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 01:53:13.479676 658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 01:53:13.479802 658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:53:13.479889 658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 01:53:13.479959 658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 01:53:13.490578 658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
I20250901 01:53:13.492354 658 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:53:13.494290 658 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 01:53:13.503379 663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:13.503567 664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:13.503935 666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:13.504598 658 server_base.cc:1047] running on GCE node
I20250901 01:53:13.505596 658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:53:13.507403 658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:53:13.508646 658 hybrid_clock.cc:648] HybridClock initialized: now 1756691593508530 us; error 128 us; skew 500 ppm
I20250901 01:53:13.509057 658 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:53:13.512020 658 webserver.cc:480] Webserver started at http://127.0.104.62:32963/ using document root <none> and password file <none>
I20250901 01:53:13.512871 658 fs_manager.cc:362] Metadata directory not provided
I20250901 01:53:13.513002 658 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 01:53:13.513382 658 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 01:53:13.516019 658 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/instance:
uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21"
format_stamp: "Formatted at 2025-09-01 01:53:13 on dist-test-slave-r4vn"
I20250901 01:53:13.516839 658 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal/instance:
uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21"
format_stamp: "Formatted at 2025-09-01 01:53:13 on dist-test-slave-r4vn"
I20250901 01:53:13.522627 658 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20250901 01:53:13.526268 672 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:13.527719 658 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250901 01:53:13.527916 658 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data,/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21"
format_stamp: "Formatted at 2025-09-01 01:53:13 on dist-test-slave-r4vn"
I20250901 01:53:13.528148 658 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:53:13.555549 658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 01:53:13.556509 658 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:53:13.556803 658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:53:13.579053 658 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35805
I20250901 01:53:13.579072 723 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35805 every 8 connection(s)
I20250901 01:53:13.580691 658 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/info.pb
I20250901 01:53:13.585191 724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:13.585129 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1fHXBS/build/asan/bin/kudu as pid 658
I20250901 01:53:13.585459 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal/instance
I20250901 01:53:13.589212 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1fHXBS/build/asan/bin/kudu
/tmp/dist-test-task1fHXBS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:33025
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250901 01:53:13.595252 724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } has no permanent_uuid. Determining permanent_uuid...
I20250901 01:53:13.613940 724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
W20250901 01:53:13.615311 675 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:33025: connect: Connection refused (error 111)
W20250901 01:53:13.617053 724 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:33025: Network error: Client connection negotiation failed: client connection to 127.0.104.61:33025: connect: Connection refused (error 111)
I20250901 01:53:13.666462 724 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } attempt: 1
W20250901 01:53:13.668629 724 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:33025: Network error: Client connection negotiation failed: client connection to 127.0.104.61:33025: connect: Connection refused (error 111)
I20250901 01:53:13.736936 724 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } attempt: 2
W20250901 01:53:13.739022 724 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:33025: Network error: Client connection negotiation failed: client connection to 127.0.104.61:33025: connect: Connection refused (error 111)
I20250901 01:53:13.830363 724 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } attempt: 3
W20250901 01:53:13.833508 724 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:33025: Network error: Client connection negotiation failed: client connection to 127.0.104.61:33025: connect: Connection refused (error 111)
W20250901 01:53:13.957366 725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:53:13.957731 725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:53:13.957875 725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 01:53:13.966718 725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 01:53:13.966832 725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:53:13.966889 725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 01:53:13.966960 725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 01:53:13.976830 724 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } attempt: 4
I20250901 01:53:13.977555 725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
W20250901 01:53:13.978832 724 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:33025: Network error: Client connection negotiation failed: client connection to 127.0.104.61:33025: connect: Connection refused (error 111)
I20250901 01:53:13.979506 725 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:53:13.981281 725 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 01:53:13.990051 733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:13.990422 734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:13.990751 736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:13.991150 725 server_base.cc:1047] running on GCE node
I20250901 01:53:13.991987 725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:53:13.993582 725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:53:13.994802 725 hybrid_clock.cc:648] HybridClock initialized: now 1756691593994752 us; error 53 us; skew 500 ppm
I20250901 01:53:13.995182 725 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:53:13.998304 725 webserver.cc:480] Webserver started at http://127.0.104.61:34857/ using document root <none> and password file <none>
I20250901 01:53:13.999166 725 fs_manager.cc:362] Metadata directory not provided
I20250901 01:53:13.999279 725 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 01:53:13.999706 725 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 01:53:14.003860 725 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/instance:
uuid: "b557230fc3b142c4bf5f6f58d241905f"
format_stamp: "Formatted at 2025-09-01 01:53:13 on dist-test-slave-r4vn"
I20250901 01:53:14.004801 725 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal/instance:
uuid: "b557230fc3b142c4bf5f6f58d241905f"
format_stamp: "Formatted at 2025-09-01 01:53:13 on dist-test-slave-r4vn"
I20250901 01:53:14.011065 725 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250901 01:53:14.014940 742 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:14.016494 725 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250901 01:53:14.016691 725 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data,/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
uuid: "b557230fc3b142c4bf5f6f58d241905f"
format_stamp: "Formatted at 2025-09-01 01:53:13 on dist-test-slave-r4vn"
I20250901 01:53:14.016913 725 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:53:14.034206 725 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 01:53:14.035212 725 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:53:14.035508 725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:53:14.058221 725 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:33025
I20250901 01:53:14.058245 793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:33025 every 8 connection(s)
I20250901 01:53:14.059837 725 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/info.pb
I20250901 01:53:14.061112 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1fHXBS/build/asan/bin/kudu as pid 725
I20250901 01:53:14.061298 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal/instance
I20250901 01:53:14.062407 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1fHXBS/build/asan/bin/kudu
/tmp/dist-test-task1fHXBS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:36985
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250901 01:53:14.064004 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:14.072902 794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } has no permanent_uuid. Determining permanent_uuid...
I20250901 01:53:14.088255 794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
I20250901 01:53:14.100725 794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } has no permanent_uuid. Determining permanent_uuid...
W20250901 01:53:14.102041 746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:53:14.104022 794 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:36985: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:14.153379 794 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } attempt: 1
W20250901 01:53:14.155643 794 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:36985: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:14.223913 794 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } attempt: 2
W20250901 01:53:14.225883 794 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:36985: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:14.278232 724 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } attempt: 5
I20250901 01:53:14.285918 724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } has no permanent_uuid. Determining permanent_uuid...
W20250901 01:53:14.287950 724 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:36985: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:14.317200 794 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } attempt: 3
W20250901 01:53:14.319594 794 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:36985: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:14.339200 724 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } attempt: 1
W20250901 01:53:14.341466 724 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:36985: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:14.409806 724 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } attempt: 2
W20250901 01:53:14.411825 724 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:36985: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:14.462889 794 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } attempt: 4
W20250901 01:53:14.465009 794 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:36985: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:53:14.472906 795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:53:14.473287 795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:53:14.473415 795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 01:53:14.482370 795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 01:53:14.482501 795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:53:14.482594 795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 01:53:14.482667 795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 01:53:14.493315 795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:36985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
I20250901 01:53:14.495107 795 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:53:14.497014 795 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 01:53:14.506413 804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:14.506820 803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:14.507136 806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:14.507946 795 server_base.cc:1047] running on GCE node
I20250901 01:53:14.508898 795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:53:14.510635 795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:53:14.511883 795 hybrid_clock.cc:648] HybridClock initialized: now 1756691594511828 us; error 61 us; skew 500 ppm
I20250901 01:53:14.512288 795 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:53:14.515216 795 webserver.cc:480] Webserver started at http://127.0.104.60:35767/ using document root <none> and password file <none>
I20250901 01:53:14.516067 795 fs_manager.cc:362] Metadata directory not provided
I20250901 01:53:14.516192 795 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 01:53:14.516587 795 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 01:53:14.518064 724 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } attempt: 3
I20250901 01:53:14.519650 795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/data/instance:
uuid: "835a442fbc4c48eb9148ca677e410a59"
format_stamp: "Formatted at 2025-09-01 01:53:14 on dist-test-slave-r4vn"
W20250901 01:53:14.520124 724 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:36985: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:14.520673 795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/wal/instance:
uuid: "835a442fbc4c48eb9148ca677e410a59"
format_stamp: "Formatted at 2025-09-01 01:53:14 on dist-test-slave-r4vn"
I20250901 01:53:14.526661 795 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250901 01:53:14.530644 812 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:14.532316 795 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250901 01:53:14.532533 795 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/data,/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/wal
uuid: "835a442fbc4c48eb9148ca677e410a59"
format_stamp: "Formatted at 2025-09-01 01:53:14 on dist-test-slave-r4vn"
I20250901 01:53:14.533296 795 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:53:14.563966 795 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 01:53:14.564889 795 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:53:14.565160 795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:53:14.593418 795 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:36985
I20250901 01:53:14.593407 863 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:36985 every 8 connection(s)
I20250901 01:53:14.595896 795 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/data/info.pb
I20250901 01:53:14.599452 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1fHXBS/build/asan/bin/kudu as pid 795
I20250901 01:53:14.599656 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/wal/instance
I20250901 01:53:14.600420 864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:14.607306 864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } has no permanent_uuid. Determining permanent_uuid...
I20250901 01:53:14.624173 864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
I20250901 01:53:14.632269 864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } has no permanent_uuid. Determining permanent_uuid...
I20250901 01:53:14.644340 864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: Bootstrap starting.
I20250901 01:53:14.648118 864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:14.649372 864 log.cc:826] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: Log is configured to *not* fsync() on all Append() calls
I20250901 01:53:14.652385 864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: No bootstrap required, opened a new log
I20250901 01:53:14.657963 864 raft_consensus.cc:357] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:14.658421 864 raft_consensus.cc:383] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:14.658537 864 raft_consensus.cc:738] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835a442fbc4c48eb9148ca677e410a59, State: Initialized, Role: FOLLOWER
I20250901 01:53:14.659272 864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:14.660272 873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:14.660529 873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:14.661614 864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 01:53:14.672278 795 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: OK, instance_id { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" instance_seqno: 1756691593572699 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250901 01:53:14.677801 795 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:53:14.680526 887 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250901 01:53:14.680713 887 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250901 01:53:14.697378 724 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } attempt: 4
I20250901 01:53:14.712358 724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Bootstrap starting.
I20250901 01:53:14.717012 724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:14.718597 724 log.cc:826] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Log is configured to *not* fsync() on all Append() calls
I20250901 01:53:14.728564 724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: No bootstrap required, opened a new log
I20250901 01:53:14.736462 724 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:14.736948 724 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:14.737094 724 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c35f5a36eeb4a93aecdd71e2d1c3a21, State: Initialized, Role: FOLLOWER
I20250901 01:53:14.738212 724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:14.748888 891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:14.749257 891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:14.750267 724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 01:53:14.764225 794 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } attempt: 5
I20250901 01:53:14.782554 794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Bootstrap starting.
I20250901 01:53:14.788256 794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:14.790045 794 log.cc:826] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Log is configured to *not* fsync() on all Append() calls
W20250901 01:53:14.809855 678 tablet.cc:2378] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 01:53:14.813086 794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: No bootstrap required, opened a new log
W20250901 01:53:14.817584 818 tablet.cc:2378] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 01:53:14.821564 794 raft_consensus.cc:357] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:14.822063 794 raft_consensus.cc:383] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:14.822203 794 raft_consensus.cc:738] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b557230fc3b142c4bf5f6f58d241905f, State: Initialized, Role: FOLLOWER
I20250901 01:53:14.823104 794 consensus_queue.cc:260] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:14.834980 906 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:14.835426 906 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:14.837256 794 sys_catalog.cc:564] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: configured and running, proceeding with master startup.
I20250901 01:53:14.844161 908 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250901 01:53:14.849416 658 master_runner.cc:186] Error getting master registration for 127.0.104.61:33025: OK, instance_id { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" instance_seqno: 1756691594051046 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250901 01:53:14.857964 658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:53:14.888402 725 master_runner.cc:263] No leader master found from master b557230fc3b142c4bf5f6f58d241905f
I20250901 01:53:14.889145 725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:53:14.893236 922 catalog_manager.cc:1560] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250901 01:53:14.893410 922 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250901 01:53:15.029003 891 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:15.029318 891 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:15.031280 891 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:15.032078 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f" is_pre_election: true
I20250901 01:53:15.032330 839 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835a442fbc4c48eb9148ca677e410a59" is_pre_election: true
I20250901 01:53:15.032632 769 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c35f5a36eeb4a93aecdd71e2d1c3a21 in term 0.
I20250901 01:53:15.032855 839 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c35f5a36eeb4a93aecdd71e2d1c3a21 in term 0.
I20250901 01:53:15.033499 675 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, b557230fc3b142c4bf5f6f58d241905f; no voters:
I20250901 01:53:15.034034 891 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:15.034241 891 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:15.034356 891 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:15.037982 891 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
W20250901 01:53:15.038511 748 tablet.cc:2378] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 01:53:15.039441 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f"
I20250901 01:53:15.039669 769 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:15.040272 839 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835a442fbc4c48eb9148ca677e410a59"
I20250901 01:53:15.040488 839 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:15.043015 769 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c35f5a36eeb4a93aecdd71e2d1c3a21 in term 1.
I20250901 01:53:15.043749 675 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, b557230fc3b142c4bf5f6f58d241905f; no voters:
I20250901 01:53:15.043984 839 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c35f5a36eeb4a93aecdd71e2d1c3a21 in term 1.
I20250901 01:53:15.044726 891 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 1 election: Requested vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:15.045014 891 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:15.045171 891 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 1 LEADER]: Becoming Leader. State: Replica: 2c35f5a36eeb4a93aecdd71e2d1c3a21, State: Running, Role: LEADER
I20250901 01:53:15.045743 891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:15.054191 925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c35f5a36eeb4a93aecdd71e2d1c3a21. Latest consensus state: current_term: 1 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:15.056056 925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:15.066001 927 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 01:53:15.081127 927 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 01:53:15.102319 769 raft_consensus.cc:1273] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 1 FOLLOWER]: Refusing update from remote peer 2c35f5a36eeb4a93aecdd71e2d1c3a21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 01:53:15.102792 839 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 1 FOLLOWER]: Refusing update from remote peer 2c35f5a36eeb4a93aecdd71e2d1c3a21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 01:53:15.103896 925 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:15.104416 891 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:15.145298 873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c35f5a36eeb4a93aecdd71e2d1c3a21. Latest consensus state: current_term: 1 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:15.145661 873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:15.160243 931 mvcc.cc:204] Tried to move back new op lower bound from 7195408773527027712 to 7195408773312868352. Current Snapshot: MvccSnapshot[applied={T|T < 7195408773527027712}]
I20250901 01:53:15.164387 906 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c35f5a36eeb4a93aecdd71e2d1c3a21. Latest consensus state: current_term: 1 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:15.164790 906 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:15.167042 891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:15.167336 891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:15.179060 873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:15.179127 925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:15.179337 873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:15.179356 925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:15.179859 930 mvcc.cc:204] Tried to move back new op lower bound from 7195408773527027712 to 7195408773312868352. Current Snapshot: MvccSnapshot[applied={T|T < 7195408773527027712}]
I20250901 01:53:15.180729 927 catalog_manager.cc:1349] Generated new cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:15.180863 927 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 01:53:15.187294 906 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:15.187633 906 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:15.234061 927 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 01:53:15.236022 927 catalog_manager.cc:1506] Loading token signing keys...
I20250901 01:53:15.257622 927 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Generated new TSK 0
I20250901 01:53:15.258911 927 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 01:53:15.260349 927 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 01:53:15.261582 927 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250901 01:53:15.271412 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1fHXBS/build/asan/bin/kudu
/tmp/dist-test-task1fHXBS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--builtin_ntp_servers=127.0.104.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250901 01:53:15.676367 938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:53:15.676710 938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:53:15.676837 938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250901 01:53:15.683135 887 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:15.683243 887 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: loading cluster ID for follower catalog manager: success
I20250901 01:53:15.685933 887 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: acquiring CA information for follower catalog manager: success
W20250901 01:53:15.686013 938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:53:15.686280 938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250901 01:53:15.687951 887 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 01:53:15.697031 938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
I20250901 01:53:15.698820 938 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:53:15.700738 938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 01:53:15.712278 944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:15.712682 943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:15.712255 946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:15.713218 938 server_base.cc:1047] running on GCE node
I20250901 01:53:15.713927 938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:53:15.715015 938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:53:15.716220 938 hybrid_clock.cc:648] HybridClock initialized: now 1756691595716174 us; error 46 us; skew 500 ppm
I20250901 01:53:15.716540 938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:53:15.718871 938 webserver.cc:480] Webserver started at http://127.0.104.1:35171/ using document root <none> and password file <none>
I20250901 01:53:15.719722 938 fs_manager.cc:362] Metadata directory not provided
I20250901 01:53:15.719841 938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 01:53:15.720222 938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 01:53:15.723026 938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/data/instance:
uuid: "b65b3154aefd49f7a37f70e9207e9853"
format_stamp: "Formatted at 2025-09-01 01:53:15 on dist-test-slave-r4vn"
I20250901 01:53:15.723893 938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/instance:
uuid: "b65b3154aefd49f7a37f70e9207e9853"
format_stamp: "Formatted at 2025-09-01 01:53:15 on dist-test-slave-r4vn"
I20250901 01:53:15.729876 938 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250901 01:53:15.733722 952 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:15.735383 938 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250901 01:53:15.735607 938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/data,/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal
uuid: "b65b3154aefd49f7a37f70e9207e9853"
format_stamp: "Formatted at 2025-09-01 01:53:15 on dist-test-slave-r4vn"
I20250901 01:53:15.735826 938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:53:15.756325 957 raft_consensus.cc:491] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:15.756428 938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 01:53:15.756518 957 raft_consensus.cc:513] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:15.757378 938 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:53:15.757861 957 leader_election.cc:290] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025)
I20250901 01:53:15.757678 938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:53:15.758478 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "835a442fbc4c48eb9148ca677e410a59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f" is_pre_election: true
I20250901 01:53:15.758584 699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "835a442fbc4c48eb9148ca677e410a59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" is_pre_election: true
I20250901 01:53:15.758731 768 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835a442fbc4c48eb9148ca677e410a59 in term 1.
I20250901 01:53:15.759295 815 leader_election.cc:304] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835a442fbc4c48eb9148ca677e410a59, b557230fc3b142c4bf5f6f58d241905f; no voters:
I20250901 01:53:15.759696 957 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250901 01:53:15.759862 957 raft_consensus.cc:491] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:15.759969 957 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 1 FOLLOWER]: Advancing to term 2
I20250901 01:53:15.761945 957 raft_consensus.cc:513] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:15.762646 957 leader_election.cc:290] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [CANDIDATE]: Term 2 election: Requested vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025)
I20250901 01:53:15.763150 699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "835a442fbc4c48eb9148ca677e410a59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21"
I20250901 01:53:15.763886 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "835a442fbc4c48eb9148ca677e410a59" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f"
I20250901 01:53:15.764039 958 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:15.764180 958 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:15.764676 768 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 1 FOLLOWER]: Advancing to term 2
I20250901 01:53:15.765651 958 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:15.766176 938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 01:53:15.766206 699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b557230fc3b142c4bf5f6f58d241905f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" is_pre_election: true
I20250901 01:53:15.766758 838 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b557230fc3b142c4bf5f6f58d241905f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "835a442fbc4c48eb9148ca677e410a59" is_pre_election: true
I20250901 01:53:15.766996 838 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b557230fc3b142c4bf5f6f58d241905f in current term 2: Already voted for candidate 835a442fbc4c48eb9148ca677e410a59 in this term.
I20250901 01:53:15.767443 746 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:15.768326 768 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835a442fbc4c48eb9148ca677e410a59 in term 2.
I20250901 01:53:15.768628 958 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250901 01:53:15.768776 815 leader_election.cc:304] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835a442fbc4c48eb9148ca677e410a59, b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21
I20250901 01:53:15.769120 938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 01:53:15.769121 957 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 2 FOLLOWER]: Leader election won for term 2
I20250901 01:53:15.769263 938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:15.769395 938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 01:53:15.769512 938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:15.769622 957 raft_consensus.cc:695] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 2 LEADER]: Becoming Leader. State: Replica: 835a442fbc4c48eb9148ca677e410a59, State: Running, Role: LEADER
I20250901 01:53:15.770340 957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:15.772019 961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 835a442fbc4c48eb9148ca677e410a59. Latest consensus state: current_term: 2 leader_uuid: "835a442fbc4c48eb9148ca677e410a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:15.772234 961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:15.772933 963 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 01:53:15.775727 963 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 01:53:15.776935 963 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:15.777038 963 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 01:53:15.778904 963 catalog_manager.cc:1506] Loading token signing keys...
I20250901 01:53:15.780740 963 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: Loaded TSK: 0
I20250901 01:53:15.782284 963 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 01:53:15.783516 963 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 01:53:15.784776 963 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250901 01:53:15.811750 938 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:45315
I20250901 01:53:15.811837 1068 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:45315 every 8 connection(s)
I20250901 01:53:15.813335 938 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/data/info.pb
I20250901 01:53:15.819250 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1fHXBS/build/asan/bin/kudu as pid 938
I20250901 01:53:15.819489 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/instance
I20250901 01:53:15.824807 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1fHXBS/build/asan/bin/kudu
/tmp/dist-test-task1fHXBS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--builtin_ntp_servers=127.0.104.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250901 01:53:15.875752 768 raft_consensus.cc:1238] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 2 FOLLOWER]: Rejecting Update request from peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 for earlier term 1. Current term is 2. Ops: []
I20250901 01:53:15.876358 838 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 2 LEADER]: Rejecting Update request from peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 for earlier term 1. Current term is 2. Ops: []
I20250901 01:53:15.877614 1080 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250901 01:53:15.877911 1080 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250901 01:53:15.878093 1080 consensus_queue.cc:1230] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Peer 835a442fbc4c48eb9148ca677e410a59 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250901 01:53:15.879974 1081 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 1 LEADER]: Stepping down as leader of term 1
I20250901 01:53:15.880232 1081 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2c35f5a36eeb4a93aecdd71e2d1c3a21, State: Running, Role: LEADER
I20250901 01:53:15.882798 1081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:15.884181 1081 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 1 FOLLOWER]: Advancing to term 2
I20250901 01:53:15.890578 1069 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35805
I20250901 01:53:15.891009 1070 heartbeater.cc:344] Connected to a master server at 127.0.104.60:36985
I20250901 01:53:15.891175 1070 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:15.891027 1069 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:15.892392 1070 heartbeater.cc:507] Master 127.0.104.60:36985 requested a full tablet report, sending...
I20250901 01:53:15.892707 1069 heartbeater.cc:507] Master 127.0.104.62:35805 requested a full tablet report, sending...
I20250901 01:53:15.894721 1071 heartbeater.cc:344] Connected to a master server at 127.0.104.61:33025
I20250901 01:53:15.894882 1071 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:15.894969 829 ts_manager.cc:194] Registered new tserver with Master: b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:15.895426 1071 heartbeater.cc:507] Master 127.0.104.61:33025 requested a full tablet report, sending...
I20250901 01:53:15.895720 922 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:15.895812 922 catalog_manager.cc:1554] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: loading cluster ID for follower catalog manager: success
I20250901 01:53:15.897207 759 ts_manager.cc:194] Registered new tserver with Master: b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:15.897886 829 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:52113
I20250901 01:53:15.899648 922 catalog_manager.cc:1576] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: acquiring CA information for follower catalog manager: success
I20250901 01:53:15.902136 922 catalog_manager.cc:1604] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 01:53:15.904335 688 ts_manager.cc:194] Registered new tserver with Master: b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:16.235672 768 raft_consensus.cc:1273] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 2 FOLLOWER]: Refusing update from remote peer 835a442fbc4c48eb9148ca677e410a59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250901 01:53:16.236500 961 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250901 01:53:16.238041 958 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 835a442fbc4c48eb9148ca677e410a59. Latest consensus state: current_term: 2 leader_uuid: "835a442fbc4c48eb9148ca677e410a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:16.238335 958 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:16.241277 957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "835a442fbc4c48eb9148ca677e410a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:16.241592 957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:16.241627 958 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "835a442fbc4c48eb9148ca677e410a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:16.241878 958 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:16.242507 699 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 2 FOLLOWER]: Refusing update from remote peer 835a442fbc4c48eb9148ca677e410a59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250901 01:53:16.243266 957 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250901 01:53:16.245561 1080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "835a442fbc4c48eb9148ca677e410a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:16.245858 1080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:16.245941 1081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 835a442fbc4c48eb9148ca677e410a59. Latest consensus state: current_term: 2 leader_uuid: "835a442fbc4c48eb9148ca677e410a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:16.246096 1081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: FOLLOWER
W20250901 01:53:16.250226 1079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:53:16.250556 1079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:53:16.250703 1079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 01:53:16.259761 1079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:53:16.259985 1079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250901 01:53:16.270468 1079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
I20250901 01:53:16.272151 1079 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:53:16.274044 1079 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 01:53:16.284231 1091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:16.284273 1090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:16.285189 1093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:16.285630 1079 server_base.cc:1047] running on GCE node
I20250901 01:53:16.286273 1079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:53:16.287272 1079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:53:16.288509 1079 hybrid_clock.cc:648] HybridClock initialized: now 1756691596288456 us; error 56 us; skew 500 ppm
I20250901 01:53:16.288839 1079 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:53:16.291260 1079 webserver.cc:480] Webserver started at http://127.0.104.2:41363/ using document root <none> and password file <none>
I20250901 01:53:16.292048 1079 fs_manager.cc:362] Metadata directory not provided
I20250901 01:53:16.292158 1079 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 01:53:16.292519 1079 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 01:53:16.295223 1079 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/data/instance:
uuid: "e3c774a4edf44d6098a2332e734967a5"
format_stamp: "Formatted at 2025-09-01 01:53:16 on dist-test-slave-r4vn"
I20250901 01:53:16.296075 1079 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal/instance:
uuid: "e3c774a4edf44d6098a2332e734967a5"
format_stamp: "Formatted at 2025-09-01 01:53:16 on dist-test-slave-r4vn"
I20250901 01:53:16.301573 1079 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250901 01:53:16.305219 1099 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:16.306670 1079 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250901 01:53:16.306842 1079 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/data,/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal
uuid: "e3c774a4edf44d6098a2332e734967a5"
format_stamp: "Formatted at 2025-09-01 01:53:16 on dist-test-slave-r4vn"
I20250901 01:53:16.307053 1079 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:53:16.345871 1079 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 01:53:16.346849 1079 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:53:16.347138 1079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:53:16.348199 1079 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 01:53:16.350502 1079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 01:53:16.350620 1079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:16.350751 1079 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 01:53:16.350826 1079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:16.391713 1079 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:35245
I20250901 01:53:16.391783 1211 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:35245 every 8 connection(s)
I20250901 01:53:16.393489 1079 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/data/info.pb
I20250901 01:53:16.395812 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1fHXBS/build/asan/bin/kudu as pid 1079
I20250901 01:53:16.396036 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal/instance
I20250901 01:53:16.402343 1070 heartbeater.cc:499] Master 127.0.104.60:36985 was elected leader, sending a full tablet report...
I20250901 01:53:16.401911 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1fHXBS/build/asan/bin/kudu
/tmp/dist-test-task1fHXBS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--builtin_ntp_servers=127.0.104.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250901 01:53:16.427321 1214 heartbeater.cc:344] Connected to a master server at 127.0.104.61:33025
I20250901 01:53:16.427815 1214 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:16.428412 1213 heartbeater.cc:344] Connected to a master server at 127.0.104.60:36985
I20250901 01:53:16.428560 1213 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:16.429154 1214 heartbeater.cc:507] Master 127.0.104.61:33025 requested a full tablet report, sending...
I20250901 01:53:16.429199 1213 heartbeater.cc:507] Master 127.0.104.60:36985 requested a full tablet report, sending...
I20250901 01:53:16.430591 1212 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35805
I20250901 01:53:16.430738 1212 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:16.431136 1212 heartbeater.cc:507] Master 127.0.104.62:35805 requested a full tablet report, sending...
I20250901 01:53:16.431370 759 ts_manager.cc:194] Registered new tserver with Master: e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:16.431746 829 ts_manager.cc:194] Registered new tserver with Master: e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:16.433069 829 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:52277
I20250901 01:53:16.433853 688 ts_manager.cc:194] Registered new tserver with Master: e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:16.745596 1081 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 835a442fbc4c48eb9148ca677e410a59)
I20250901 01:53:16.745874 1081 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:16.746817 1081 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:16.746846 957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "835a442fbc4c48eb9148ca677e410a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:16.747171 957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:16.747807 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f" is_pre_election: true
I20250901 01:53:16.750272 838 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "835a442fbc4c48eb9148ca677e410a59" is_pre_election: true
I20250901 01:53:16.751128 676 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21; no voters: 835a442fbc4c48eb9148ca677e410a59, b557230fc3b142c4bf5f6f58d241905f
I20250901 01:53:16.751693 1081 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250901 01:53:16.803784 1222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:53:16.804144 1222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:53:16.804281 1222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 01:53:16.813685 1222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:53:16.813959 1222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250901 01:53:16.826426 1222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
I20250901 01:53:16.828156 1222 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:53:16.830075 1222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 01:53:16.840751 1233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:16.841048 1230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:16.840751 1231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:16.843111 1222 server_base.cc:1047] running on GCE node
I20250901 01:53:16.844573 1222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:53:16.847185 908 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 01:53:16.847590 1222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:53:16.848834 1222 hybrid_clock.cc:648] HybridClock initialized: now 1756691596848766 us; error 71 us; skew 500 ppm
I20250901 01:53:16.849224 1222 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:53:16.852173 1222 webserver.cc:480] Webserver started at http://127.0.104.3:44631/ using document root <none> and password file <none>
I20250901 01:53:16.853092 1222 fs_manager.cc:362] Metadata directory not provided
I20250901 01:53:16.853202 1222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 01:53:16.853575 1222 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 01:53:16.856798 1222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/data/instance:
uuid: "ad2ba2450480454388723cd729cc438e"
format_stamp: "Formatted at 2025-09-01 01:53:16 on dist-test-slave-r4vn"
I20250901 01:53:16.857764 1222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal/instance:
uuid: "ad2ba2450480454388723cd729cc438e"
format_stamp: "Formatted at 2025-09-01 01:53:16 on dist-test-slave-r4vn"
I20250901 01:53:16.864114 1222 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20250901 01:53:16.868404 1239 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:16.870085 1222 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250901 01:53:16.870312 1222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/data,/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal
uuid: "ad2ba2450480454388723cd729cc438e"
format_stamp: "Formatted at 2025-09-01 01:53:16 on dist-test-slave-r4vn"
I20250901 01:53:16.870565 1222 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:53:16.904036 1222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 01:53:16.905109 1222 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:53:16.905494 1222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:53:16.906639 1222 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 01:53:16.910039 1222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 01:53:16.910179 1222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:16.910303 1222 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 01:53:16.910392 1222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:16.935708 1213 heartbeater.cc:499] Master 127.0.104.60:36985 was elected leader, sending a full tablet report...
I20250901 01:53:16.950709 1222 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:42927
I20250901 01:53:16.950773 1351 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:42927 every 8 connection(s)
I20250901 01:53:16.952205 1222 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/data/info.pb
I20250901 01:53:16.956753 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1fHXBS/build/asan/bin/kudu as pid 1222
I20250901 01:53:16.956967 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal/instance
I20250901 01:53:16.986443 1354 heartbeater.cc:344] Connected to a master server at 127.0.104.61:33025
I20250901 01:53:16.986896 1352 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35805
I20250901 01:53:16.986917 1354 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:16.987062 1353 heartbeater.cc:344] Connected to a master server at 127.0.104.60:36985
I20250901 01:53:16.987097 1352 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:16.987247 1353 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:16.988184 1354 heartbeater.cc:507] Master 127.0.104.61:33025 requested a full tablet report, sending...
I20250901 01:53:16.988240 1353 heartbeater.cc:507] Master 127.0.104.60:36985 requested a full tablet report, sending...
I20250901 01:53:16.988287 1352 heartbeater.cc:507] Master 127.0.104.62:35805 requested a full tablet report, sending...
I20250901 01:53:16.990104 759 ts_manager.cc:194] Registered new tserver with Master: ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:16.990103 829 ts_manager.cc:194] Registered new tserver with Master: ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:16.990856 688 ts_manager.cc:194] Registered new tserver with Master: ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:16.991082 829 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:43729
I20250901 01:53:17.000185 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250901 01:53:17.036772 828 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41280:
name: "default.Table_7722e49197be4ca9b9c7e67052f07c4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.037139 829 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41280:
name: "default.Table_c2864548fe1a4719b73cf55d0933e538"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.037828 827 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41280:
name: "default.Table_795876dee30e438da68abc8a5bbae13f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.038560 826 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41280:
name: "default.Table_fe15d1a454744caa9805ce4ffefdfd9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.039175 825 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41280:
name: "default.Table_476c1fc1206b4426855b5602d4b0cc88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.039652 824 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41280:
name: "default.Table_e628b8e1d2214e458c8a283659d45a07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.040133 823 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41280:
name: "default.Table_832e2227a6314ac2a050f76013e116fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.040616 822 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41280:
name: "default.Table_fee1d3b07998429a91772db3deb1ea39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.041630 821 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41280:
name: "default.Table_6bcc2bc2c6574eca8387f75fc422beb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.041875 820 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41280:
name: "default.Table_90a9fd7ee57d49b9ae8bb16bbbe41f50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:53:17.054607 827 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_795876dee30e438da68abc8a5bbae13f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:17.055295 826 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fe15d1a454744caa9805ce4ffefdfd9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:17.058028 825 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_476c1fc1206b4426855b5602d4b0cc88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:17.059098 824 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e628b8e1d2214e458c8a283659d45a07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:17.060073 823 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_832e2227a6314ac2a050f76013e116fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:17.061002 822 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fee1d3b07998429a91772db3deb1ea39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:17.062335 829 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c2864548fe1a4719b73cf55d0933e538 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:53:17.075294 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task1fHXBS/build/asan/bin/kudu pid 795
I20250901 01:53:17.076604 859 generic_service.cc:196] Checking for leaks (request via RPC)
I20250901 01:53:17.145154 1402 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 835a442fbc4c48eb9148ca677e410a59)
I20250901 01:53:17.145445 1402 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:17.146340 1402 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:17.146940 699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b557230fc3b142c4bf5f6f58d241905f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" is_pre_election: true
I20250901 01:53:17.228397 838 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b557230fc3b142c4bf5f6f58d241905f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "835a442fbc4c48eb9148ca677e410a59" is_pre_election: true
I20250901 01:53:17.229830 746 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:17.230361 1402 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250901 01:53:17.230952 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1fHXBS/build/asan/bin/kudu with pid 795
W20250901 01:53:17.256282 1365 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.60:36985)
W20250901 01:53:17.260167 1367 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 01:53:17.257810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:36985 (local address 127.0.0.1:41284)
0901 01:53:17.258117 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0901 01:53:17.258135 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0901 01:53:17.258345 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 01:53:17.258674 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":92}
W20250901 01:53:17.412010 1070 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:36985 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:53:17.440559 1213 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:36985 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:17.457516 1405 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 835a442fbc4c48eb9148ca677e410a59)
I20250901 01:53:17.457719 1405 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:17.458416 1405 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:17.458863 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f" is_pre_election: true
I20250901 01:53:17.459112 768 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c35f5a36eeb4a93aecdd71e2d1c3a21 in term 2.
I20250901 01:53:17.459507 675 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, b557230fc3b142c4bf5f6f58d241905f; no voters:
I20250901 01:53:17.459802 1405 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250901 01:53:17.459995 1405 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 835a442fbc4c48eb9148ca677e410a59)
I20250901 01:53:17.460124 1405 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 2 FOLLOWER]: Advancing to term 3
W20250901 01:53:17.460950 676 leader_election.cc:336] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:17.462360 1405 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:17.463022 1405 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 3 election: Requested vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:17.463490 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f"
I20250901 01:53:17.463685 768 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 2 FOLLOWER]: Advancing to term 3
I20250901 01:53:17.465590 768 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c35f5a36eeb4a93aecdd71e2d1c3a21 in term 3.
I20250901 01:53:17.466142 675 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, b557230fc3b142c4bf5f6f58d241905f; no voters:
I20250901 01:53:17.466573 1405 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 3 FOLLOWER]: Leader election won for term 3
W20250901 01:53:17.467561 676 leader_election.cc:336] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:17.468206 1405 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 3 LEADER]: Becoming Leader. State: Replica: 2c35f5a36eeb4a93aecdd71e2d1c3a21, State: Running, Role: LEADER
I20250901 01:53:17.468544 1405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:17.469954 1408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c35f5a36eeb4a93aecdd71e2d1c3a21. Latest consensus state: current_term: 3 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:17.470357 1408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:17.471017 1410 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 01:53:17.472914 1410 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 01:53:17.474221 1410 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:17.474398 1410 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 01:53:17.476018 1410 catalog_manager.cc:1506] Loading token signing keys...
I20250901 01:53:17.477242 1410 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Loaded TSK: 0
I20250901 01:53:17.478174 1410 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 01:53:17.479246 1410 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 01:53:17.480305 1410 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250901 01:53:17.493285 1353 heartbeater.cc:499] Master 127.0.104.60:36985 was elected leader, sending a full tablet report...
I20250901 01:53:17.501260 687 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_fee1d3b07998429a91772db3deb1ea39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.501708 689 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_6bcc2bc2c6574eca8387f75fc422beb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.502406 686 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_832e2227a6314ac2a050f76013e116fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.502470 685 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_7722e49197be4ca9b9c7e67052f07c4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.503115 684 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_c2864548fe1a4719b73cf55d0933e538"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.503160 683 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_795876dee30e438da68abc8a5bbae13f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.503846 680 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_e628b8e1d2214e458c8a283659d45a07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.503902 681 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_476c1fc1206b4426855b5602d4b0cc88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.504518 682 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_fe15d1a454744caa9805ce4ffefdfd9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:17.505065 688 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_90a9fd7ee57d49b9ae8bb16bbbe41f50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:53:17.506238 1353 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:36985 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:17.558758 768 raft_consensus.cc:1273] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 3 FOLLOWER]: Refusing update from remote peer 2c35f5a36eeb4a93aecdd71e2d1c3a21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20250901 01:53:17.559551 1405 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250901 01:53:17.560385 676 consensus_peers.cc:489] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 -> Peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Couldn't send request to peer 835a442fbc4c48eb9148ca677e410a59. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:53:17.570868 1402 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c35f5a36eeb4a93aecdd71e2d1c3a21. Latest consensus state: current_term: 3 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:17.571128 1402 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:17.573129 1408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:17.573359 1408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:17.573678 1412 mvcc.cc:204] Tried to move back new op lower bound from 7195408783591473152 to 7195408783233470464. Current Snapshot: MvccSnapshot[applied={T|T < 7195408783591473152}]
I20250901 01:53:17.575560 1402 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:17.575781 1402 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
W20250901 01:53:17.577214 687 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fee1d3b07998429a91772db3deb1ea39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:17.581758 682 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fe15d1a454744caa9805ce4ffefdfd9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:17.581873 688 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:17.582513 689 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6bcc2bc2c6574eca8387f75fc422beb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:17.583596 680 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e628b8e1d2214e458c8a283659d45a07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:17.584520 681 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_476c1fc1206b4426855b5602d4b0cc88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:17.585433 683 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_795876dee30e438da68abc8a5bbae13f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:17.586365 684 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c2864548fe1a4719b73cf55d0933e538 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:17.587324 686 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_832e2227a6314ac2a050f76013e116fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:17.588189 685 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7722e49197be4ca9b9c7e67052f07c4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:53:17.634100 687 catalog_manager.cc:2149] created HMS catalog entry for table [id=b449add8498f4d1fbb05b28cc933c647]
I20250901 01:53:17.645748 687 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_fee1d3b07998429a91772db3deb1ea39" } new_table_name: "default.Table_d1c05496169b472fa9b021013c4afd01" modify_external_catalogs: true
I20250901 01:53:17.686774 1287 tablet_service.cc:1468] Processing CreateTablet for tablet a2d3b7c130694ac6aa4249091c2c479b (DEFAULT_TABLE table=default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:17.688666 1287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d3b7c130694ac6aa4249091c2c479b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:17.700824 1147 tablet_service.cc:1468] Processing CreateTablet for tablet a2d3b7c130694ac6aa4249091c2c479b (DEFAULT_TABLE table=default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:17.700819 1004 tablet_service.cc:1468] Processing CreateTablet for tablet a2d3b7c130694ac6aa4249091c2c479b (DEFAULT_TABLE table=default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:17.701699 688 catalog_manager.cc:2149] created HMS catalog entry for table [id=39123cfe4f62420681f8d679d2d376b8]
I20250901 01:53:17.702497 1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d3b7c130694ac6aa4249091c2c479b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:17.705268 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d3b7c130694ac6aa4249091c2c479b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:17.716945 688 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_90a9fd7ee57d49b9ae8bb16bbbe41f50" } new_table_name: "default.Table_8f21a26e011c4ed1b1ae7cb364e09195" modify_external_catalogs: true
I20250901 01:53:17.720829 1146 tablet_service.cc:1468] Processing CreateTablet for tablet cfb2dd7459a94484b9df36a853de4a57 (DEFAULT_TABLE table=default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:17.720830 1286 tablet_service.cc:1468] Processing CreateTablet for tablet cfb2dd7459a94484b9df36a853de4a57 (DEFAULT_TABLE table=default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:17.721370 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb2dd7459a94484b9df36a853de4a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:17.721370 1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb2dd7459a94484b9df36a853de4a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:17.728492 1422 tablet_bootstrap.cc:492] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:17.728641 1003 tablet_service.cc:1468] Processing CreateTablet for tablet cfb2dd7459a94484b9df36a853de4a57 (DEFAULT_TABLE table=default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:17.729195 1003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb2dd7459a94484b9df36a853de4a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:17.733191 1422 tablet_bootstrap.cc:654] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:17.739058 1422 log.cc:826] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5: Log is configured to *not* fsync() on all Append() calls
I20250901 01:53:17.740675 1423 tablet_bootstrap.cc:492] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:17.744944 1424 tablet_bootstrap.cc:492] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:17.745658 1423 tablet_bootstrap.cc:654] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:17.747224 1423 log.cc:826] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853: Log is configured to *not* fsync() on all Append() calls
I20250901 01:53:17.748958 1424 tablet_bootstrap.cc:654] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:17.750912 1424 log.cc:826] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e: Log is configured to *not* fsync() on all Append() calls
I20250901 01:53:17.757341 1422 tablet_bootstrap.cc:492] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:17.757813 1422 ts_tablet_manager.cc:1397] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.000s
I20250901 01:53:17.758754 1423 tablet_bootstrap.cc:492] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:17.759079 1423 ts_tablet_manager.cc:1397] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.004s
I20250901 01:53:17.766862 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1fHXBS/build/asan/bin/kudu
/tmp/dist-test-task1fHXBS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:36985
--webserver_interface=127.0.104.60
--webserver_port=35767
--builtin_ntp_servers=127.0.104.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250901 01:53:17.769186 1424 tablet_bootstrap.cc:492] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:17.769551 1424 ts_tablet_manager.cc:1397] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.006s
I20250901 01:53:17.771071 1423 raft_consensus.cc:357] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:17.771531 1423 raft_consensus.cc:383] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:17.771672 1423 raft_consensus.cc:738] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:17.774500 1422 raft_consensus.cc:357] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:17.775028 1422 raft_consensus.cc:383] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:17.775163 1422 raft_consensus.cc:738] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:17.776067 1422 consensus_queue.cc:260] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:17.778667 1424 raft_consensus.cc:357] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:17.779100 1424 raft_consensus.cc:383] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:17.779229 1424 raft_consensus.cc:738] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:17.780150 1423 consensus_queue.cc:260] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:17.783965 1424 consensus_queue.cc:260] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:17.786908 1423 ts_tablet_manager.cc:1428] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.028s user 0.014s sys 0.000s
I20250901 01:53:17.791894 1422 ts_tablet_manager.cc:1428] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.034s user 0.007s sys 0.017s
I20250901 01:53:17.791937 1423 tablet_bootstrap.cc:492] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:17.793849 1423 tablet_bootstrap.cc:654] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:17.795563 1352 heartbeater.cc:499] Master 127.0.104.62:35805 was elected leader, sending a full tablet report...
I20250901 01:53:17.797156 1422 tablet_bootstrap.cc:492] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:17.799446 1422 tablet_bootstrap.cc:654] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:17.802034 1424 ts_tablet_manager.cc:1428] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.032s user 0.008s sys 0.004s
I20250901 01:53:17.804008 1423 tablet_bootstrap.cc:492] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:17.804265 1423 ts_tablet_manager.cc:1397] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20250901 01:53:17.805404 1424 tablet_bootstrap.cc:492] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:17.805797 1423 raft_consensus.cc:357] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:17.806100 1423 raft_consensus.cc:383] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:17.806216 1423 raft_consensus.cc:738] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:17.806560 1423 consensus_queue.cc:260] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:17.807121 1423 ts_tablet_manager.cc:1428] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250901 01:53:17.807709 1422 tablet_bootstrap.cc:492] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:17.807886 1422 ts_tablet_manager.cc:1397] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20250901 01:53:17.808808 1422 raft_consensus.cc:357] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:17.809017 1422 raft_consensus.cc:383] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:17.809091 1422 raft_consensus.cc:738] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:17.809366 1422 consensus_queue.cc:260] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:17.812819 1422 ts_tablet_manager.cc:1428] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250901 01:53:17.814199 1430 raft_consensus.cc:491] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:17.814536 1430 raft_consensus.cc:513] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
W20250901 01:53:17.818689 1072 tablet.cc:2378] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 01:53:17.827883 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfb2dd7459a94484b9df36a853de4a57" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:17.828435 1167 raft_consensus.cc:2466] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:17.829401 953 leader_election.cc:304] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:17.837312 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfb2dd7459a94484b9df36a853de4a57" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
W20250901 01:53:17.839071 954 leader_election.cc:343] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2ba2450480454388723cd729cc438e (127.0.104.3:42927): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:53:17.839656 1430 leader_election.cc:290] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:17.840440 1430 raft_consensus.cc:2802] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:17.840701 1430 raft_consensus.cc:491] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:17.840906 1430 raft_consensus.cc:3058] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:17.841943 1424 tablet_bootstrap.cc:654] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:17.845820 1430 raft_consensus.cc:513] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:17.847589 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfb2dd7459a94484b9df36a853de4a57" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
W20250901 01:53:17.848379 954 leader_election.cc:343] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ad2ba2450480454388723cd729cc438e (127.0.104.3:42927): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:53:17.848685 1430 leader_election.cc:290] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Requested vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:17.850548 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfb2dd7459a94484b9df36a853de4a57" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:17.850806 1167 raft_consensus.cc:3058] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:17.851338 1424 tablet_bootstrap.cc:492] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:17.851526 1424 ts_tablet_manager.cc:1397] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20250901 01:53:17.852592 1424 raft_consensus.cc:357] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:17.852806 1424 raft_consensus.cc:383] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:17.852885 1424 raft_consensus.cc:738] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:17.853185 1424 consensus_queue.cc:260] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:17.854621 1167 raft_consensus.cc:2466] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:17.855320 953 leader_election.cc:304] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters: ad2ba2450480454388723cd729cc438e
I20250901 01:53:17.857214 1424 ts_tablet_manager.cc:1428] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250901 01:53:17.857533 1434 raft_consensus.cc:2802] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:17.857764 1434 raft_consensus.cc:695] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Becoming Leader. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Running, Role: LEADER
I20250901 01:53:17.858408 1434 consensus_queue.cc:237] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
W20250901 01:53:17.898571 1215 tablet.cc:2378] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250901 01:53:17.955713 1355 tablet.cc:2378] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e: Can't schedule compaction. Clean time has not been advanced past its initial value.
01:53:17.967 [HMS - ERROR - pool-6-thread-2] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_fe15d1a454744caa9805ce4ffefdfd9c already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20250901 01:53:17.969650 682 catalog_manager.cc:2145] Already present: failed to create HMS catalog entry for table [id=027c4a656fba4f2186906912d4e65299]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_fe15d1a454744caa9805ce4ffefdfd9c already exists)
I20250901 01:53:17.973611 1069 heartbeater.cc:499] Master 127.0.104.62:35805 was elected leader, sending a full tablet report...
I20250901 01:53:17.975134 1212 heartbeater.cc:499] Master 127.0.104.62:35805 was elected leader, sending a full tablet report...
I20250901 01:53:17.979369 682 catalog_manager.cc:5582] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 reported cstate change: term changed from 0 to 1, leader changed from <none> to b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b65b3154aefd49f7a37f70e9207e9853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } health_report { overall_health: UNKNOWN } } }
I20250901 01:53:17.999859 682 master_service.cc:948] ReplaceTablet: received request to replace tablet cfb2dd7459a94484b9df36a853de4a57 from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:18.009701 682 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Sending DeleteTablet for 3 replicas of tablet cfb2dd7459a94484b9df36a853de4a57
I20250901 01:53:18.011111 1004 tablet_service.cc:1515] Processing DeleteTablet for tablet cfb2dd7459a94484b9df36a853de4a57 with delete_type TABLET_DATA_DELETED (replaced by tablet c6380a3fefba4c4caed049b434425733) from {username='slave'} at 127.0.0.1:35622
I20250901 01:53:18.012379 682 catalog_manager.cc:6652] ReplaceTablet: tablet cfb2dd7459a94484b9df36a853de4a57 deleted and replaced by tablet c6380a3fefba4c4caed049b434425733
I20250901 01:53:18.012835 682 master_service.cc:948] ReplaceTablet: received request to replace tablet cfb2dd7459a94484b9df36a853de4a57 from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:18.013535 682 master_service.cc:948] ReplaceTablet: received request to replace tablet a2d3b7c130694ac6aa4249091c2c479b from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:18.014842 1438 tablet_replica.cc:331] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853: stopping tablet replica
I20250901 01:53:18.015167 1147 tablet_service.cc:1515] Processing DeleteTablet for tablet cfb2dd7459a94484b9df36a853de4a57 with delete_type TABLET_DATA_DELETED (replaced by tablet c6380a3fefba4c4caed049b434425733) from {username='slave'} at 127.0.0.1:53914
I20250901 01:53:18.015326 1438 raft_consensus.cc:2241] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:53:18.015969 1438 pending_rounds.cc:70] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853: Trying to abort 1 pending ops.
I20250901 01:53:18.015986 1287 tablet_service.cc:1515] Processing DeleteTablet for tablet cfb2dd7459a94484b9df36a853de4a57 with delete_type TABLET_DATA_DELETED (replaced by tablet c6380a3fefba4c4caed049b434425733) from {username='slave'} at 127.0.0.1:58732
I20250901 01:53:18.016129 1438 pending_rounds.cc:77] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408784860229632 op_type: NO_OP noop_request { }
I20250901 01:53:18.016381 1438 raft_consensus.cc:2887] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:53:18.016520 1438 raft_consensus.cc:2270] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:18.022284 1286 tablet_service.cc:1468] Processing CreateTablet for tablet c6380a3fefba4c4caed049b434425733 (DEFAULT_TABLE table=default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.022487 1004 tablet_service.cc:1468] Processing CreateTablet for tablet c6380a3fefba4c4caed049b434425733 (DEFAULT_TABLE table=default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.022773 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6380a3fefba4c4caed049b434425733. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.022948 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6380a3fefba4c4caed049b434425733. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.023415 689 catalog_manager.cc:2149] created HMS catalog entry for table [id=8ee4e0d7bca3410b9967df0dacf8614d]
I20250901 01:53:18.027026 1438 ts_tablet_manager.cc:1905] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:18.029229 1424 tablet_bootstrap.cc:492] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:18.030380 1147 tablet_service.cc:1468] Processing CreateTablet for tablet c6380a3fefba4c4caed049b434425733 (DEFAULT_TABLE table=default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.030890 1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6380a3fefba4c4caed049b434425733. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.031034 1424 tablet_bootstrap.cc:654] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.034096 682 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Sending DeleteTablet for 3 replicas of tablet a2d3b7c130694ac6aa4249091c2c479b
I20250901 01:53:18.034818 682 catalog_manager.cc:6652] ReplaceTablet: tablet a2d3b7c130694ac6aa4249091c2c479b deleted and replaced by tablet a3c5c349f8f8435f8b125a6d527f970a
I20250901 01:53:18.035091 682 master_service.cc:948] ReplaceTablet: received request to replace tablet cfb2dd7459a94484b9df36a853de4a57 from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:18.035682 1285 tablet_service.cc:1515] Processing DeleteTablet for tablet a2d3b7c130694ac6aa4249091c2c479b with delete_type TABLET_DATA_DELETED (replaced by tablet a3c5c349f8f8435f8b125a6d527f970a) from {username='slave'} at 127.0.0.1:58732
I20250901 01:53:18.035823 689 master_service.cc:948] ReplaceTablet: received request to replace tablet cfb2dd7459a94484b9df36a853de4a57 from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:18.036520 1003 tablet_service.cc:1515] Processing DeleteTablet for tablet a2d3b7c130694ac6aa4249091c2c479b with delete_type TABLET_DATA_DELETED (replaced by tablet a3c5c349f8f8435f8b125a6d527f970a) from {username='slave'} at 127.0.0.1:35622
I20250901 01:53:18.036753 1146 tablet_service.cc:1515] Processing DeleteTablet for tablet a2d3b7c130694ac6aa4249091c2c479b with delete_type TABLET_DATA_DELETED (replaced by tablet a3c5c349f8f8435f8b125a6d527f970a) from {username='slave'} at 127.0.0.1:53914
I20250901 01:53:18.040340 1438 ts_tablet_manager.cc:1918] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:18.040887 1438 log.cc:1199] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/wals/cfb2dd7459a94484b9df36a853de4a57
I20250901 01:53:18.041532 1438 ts_tablet_manager.cc:1939] T cfb2dd7459a94484b9df36a853de4a57 P b65b3154aefd49f7a37f70e9207e9853: Deleting consensus metadata
I20250901 01:53:18.043540 689 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_61dfd07a8f784854b5d9e104e010daaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:18.045468 1423 tablet_bootstrap.cc:492] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:18.047055 682 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_c95fbf8fa7a64aa18002f1e268cdea68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:18.047345 1423 tablet_bootstrap.cc:654] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.052026 1438 tablet_replica.cc:331] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853: stopping tablet replica
I20250901 01:53:18.052139 676 catalog_manager.cc:4928] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): tablet cfb2dd7459a94484b9df36a853de4a57 (table default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]) successfully deleted
I20250901 01:53:18.052714 1438 raft_consensus.cc:2241] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:18.053030 1438 raft_consensus.cc:2270] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:18.053647 1424 tablet_bootstrap.cc:492] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:18.053813 1424 ts_tablet_manager.cc:1397] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.003s
I20250901 01:53:18.054207 1438 ts_tablet_manager.cc:1905] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:18.054699 1424 raft_consensus.cc:357] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.054895 1424 raft_consensus.cc:383] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.054991 1424 raft_consensus.cc:738] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.055276 1424 consensus_queue.cc:260] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.060819 1003 tablet_service.cc:1468] Processing CreateTablet for tablet 6263cf06150b49fbbda6c89bd1fb979e (DEFAULT_TABLE table=default.table_6bcc2bc2c6574eca8387f75fc422beb9 [id=8ee4e0d7bca3410b9967df0dacf8614d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.061029 1440 tablet_replica.cc:331] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e: stopping tablet replica
I20250901 01:53:18.061285 1003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6263cf06150b49fbbda6c89bd1fb979e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.061543 1440 raft_consensus.cc:2241] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:18.061998 1440 raft_consensus.cc:2270] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:18.062139 1423 tablet_bootstrap.cc:492] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:18.062398 1423 ts_tablet_manager.cc:1397] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20250901 01:53:18.063344 1423 raft_consensus.cc:357] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.063612 1423 raft_consensus.cc:383] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.063772 1423 raft_consensus.cc:738] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.064076 1423 consensus_queue.cc:260] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.064669 1423 ts_tablet_manager.cc:1428] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:53:18.065122 1004 tablet_service.cc:1468] Processing CreateTablet for tablet a3c5c349f8f8435f8b125a6d527f970a (DEFAULT_TABLE table=default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.068141 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c5c349f8f8435f8b125a6d527f970a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.068394 1422 tablet_bootstrap.cc:492] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:18.069135 1438 ts_tablet_manager.cc:1918] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:53:18.069348 1438 log.cc:1199] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/wals/a2d3b7c130694ac6aa4249091c2c479b
I20250901 01:53:18.070225 1438 ts_tablet_manager.cc:1939] T a2d3b7c130694ac6aa4249091c2c479b P b65b3154aefd49f7a37f70e9207e9853: Deleting consensus metadata
I20250901 01:53:18.070978 1285 tablet_service.cc:1468] Processing CreateTablet for tablet a3c5c349f8f8435f8b125a6d527f970a (DEFAULT_TABLE table=default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.071466 676 catalog_manager.cc:4928] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): tablet a2d3b7c130694ac6aa4249091c2c479b (table default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647]) successfully deleted
I20250901 01:53:18.071573 1285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c5c349f8f8435f8b125a6d527f970a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.073078 1146 tablet_service.cc:1468] Processing CreateTablet for tablet a3c5c349f8f8435f8b125a6d527f970a (DEFAULT_TABLE table=default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.073169 1286 tablet_service.cc:1468] Processing CreateTablet for tablet 6263cf06150b49fbbda6c89bd1fb979e (DEFAULT_TABLE table=default.table_6bcc2bc2c6574eca8387f75fc422beb9 [id=8ee4e0d7bca3410b9967df0dacf8614d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.073628 1147 tablet_service.cc:1468] Processing CreateTablet for tablet 6263cf06150b49fbbda6c89bd1fb979e (DEFAULT_TABLE table=default.table_6bcc2bc2c6574eca8387f75fc422beb9 [id=8ee4e0d7bca3410b9967df0dacf8614d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.073958 1422 tablet_bootstrap.cc:654] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.075863 1440 ts_tablet_manager.cc:1905] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:18.078289 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6263cf06150b49fbbda6c89bd1fb979e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.086432 1423 tablet_bootstrap.cc:492] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:18.088428 1423 tablet_bootstrap.cc:654] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.088996 1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c5c349f8f8435f8b125a6d527f970a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.094542 1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6263cf06150b49fbbda6c89bd1fb979e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.099246 1439 tablet_replica.cc:331] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5: stopping tablet replica
I20250901 01:53:18.100070 1439 raft_consensus.cc:2241] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:18.102283 1439 raft_consensus.cc:2270] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:18.103539 1424 ts_tablet_manager.cc:1428] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.050s user 0.003s sys 0.000s
I20250901 01:53:18.105492 1439 ts_tablet_manager.cc:1905] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:18.105763 680 catalog_manager.cc:2149] created HMS catalog entry for table [id=f1db819ca90642eaac30e77fd5f5c8d5]
I20250901 01:53:18.112855 1423 tablet_bootstrap.cc:492] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:18.113072 1423 ts_tablet_manager.cc:1397] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250901 01:53:18.114007 1423 raft_consensus.cc:357] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.114284 1423 raft_consensus.cc:383] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.114365 1423 raft_consensus.cc:738] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.114614 1423 consensus_queue.cc:260] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.115430 1423 ts_tablet_manager.cc:1428] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:53:18.115758 1423 tablet_bootstrap.cc:492] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:18.117457 1423 tablet_bootstrap.cc:654] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.123185 1440 ts_tablet_manager.cc:1918] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:53:18.123410 1440 log.cc:1199] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal/wals/cfb2dd7459a94484b9df36a853de4a57
I20250901 01:53:18.123853 1440 ts_tablet_manager.cc:1939] T cfb2dd7459a94484b9df36a853de4a57 P ad2ba2450480454388723cd729cc438e: Deleting consensus metadata
I20250901 01:53:18.125558 1440 tablet_replica.cc:331] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e: stopping tablet replica
I20250901 01:53:18.125665 674 catalog_manager.cc:4928] TS ad2ba2450480454388723cd729cc438e (127.0.104.3:42927): tablet cfb2dd7459a94484b9df36a853de4a57 (table default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]) successfully deleted
I20250901 01:53:18.125775 1440 raft_consensus.cc:2241] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:18.125941 1440 raft_consensus.cc:2270] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:18.129319 1440 ts_tablet_manager.cc:1905] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:18.133257 1423 tablet_bootstrap.cc:492] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:18.133483 1423 ts_tablet_manager.cc:1397] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250901 01:53:18.134495 1423 raft_consensus.cc:357] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.134701 1423 raft_consensus.cc:383] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.134799 1423 raft_consensus.cc:738] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.135035 1423 consensus_queue.cc:260] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.136935 1004 tablet_service.cc:1468] Processing CreateTablet for tablet 9cd7993f6a99468caa9b13fc4e8c0ba1 (DEFAULT_TABLE table=default.table_e628b8e1d2214e458c8a283659d45a07 [id=f1db819ca90642eaac30e77fd5f5c8d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.137441 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cd7993f6a99468caa9b13fc4e8c0ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.137356 1286 tablet_service.cc:1468] Processing CreateTablet for tablet 9cd7993f6a99468caa9b13fc4e8c0ba1 (DEFAULT_TABLE table=default.table_e628b8e1d2214e458c8a283659d45a07 [id=f1db819ca90642eaac30e77fd5f5c8d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.137802 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cd7993f6a99468caa9b13fc4e8c0ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.139384 680 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_5178a246373c4a26a1a2e11a77c7e56c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:18.140151 1424 tablet_bootstrap.cc:492] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:18.142108 1424 tablet_bootstrap.cc:654] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.142843 1423 ts_tablet_manager.cc:1428] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250901 01:53:18.107609 1422 tablet_bootstrap.cc:492] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:18.143612 1422 ts_tablet_manager.cc:1397] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.002s
I20250901 01:53:18.144620 1422 raft_consensus.cc:357] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.144827 1422 raft_consensus.cc:383] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.144935 1422 raft_consensus.cc:738] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.145202 1422 consensus_queue.cc:260] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.110760 1429 raft_consensus.cc:491] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:18.145872 1429 raft_consensus.cc:513] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:18.147922 1429 leader_election.cc:290] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2ba2450480454388723cd729cc438e (127.0.104.3:42927), b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:18.137367 1147 tablet_service.cc:1468] Processing CreateTablet for tablet 9cd7993f6a99468caa9b13fc4e8c0ba1 (DEFAULT_TABLE table=default.table_e628b8e1d2214e458c8a283659d45a07 [id=f1db819ca90642eaac30e77fd5f5c8d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.149755 1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cd7993f6a99468caa9b13fc4e8c0ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.151882 1440 ts_tablet_manager.cc:1918] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:53:18.152078 1440 log.cc:1199] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal/wals/a2d3b7c130694ac6aa4249091c2c479b
I20250901 01:53:18.152490 1440 ts_tablet_manager.cc:1939] T a2d3b7c130694ac6aa4249091c2c479b P ad2ba2450480454388723cd729cc438e: Deleting consensus metadata
I20250901 01:53:18.153578 674 catalog_manager.cc:4928] TS ad2ba2450480454388723cd729cc438e (127.0.104.3:42927): tablet a2d3b7c130694ac6aa4249091c2c479b (table default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647]) successfully deleted
I20250901 01:53:18.165338 1423 tablet_bootstrap.cc:492] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:18.167418 1423 tablet_bootstrap.cc:654] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.186376 1422 ts_tablet_manager.cc:1428] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.043s user 0.004s sys 0.007s
I20250901 01:53:18.187011 1422 tablet_bootstrap.cc:492] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:18.188902 1422 tablet_bootstrap.cc:654] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.192242 1423 tablet_bootstrap.cc:492] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:18.192462 1423 ts_tablet_manager.cc:1397] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.004s
I20250901 01:53:18.192795 1424 tablet_bootstrap.cc:492] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:18.193025 1424 ts_tablet_manager.cc:1397] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.001s
I20250901 01:53:18.193339 1423 raft_consensus.cc:357] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.193573 1423 raft_consensus.cc:383] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.193661 1423 raft_consensus.cc:738] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.194077 1424 raft_consensus.cc:357] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.194388 1424 raft_consensus.cc:383] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.194514 1424 raft_consensus.cc:738] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.195204 1424 consensus_queue.cc:260] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.195195 1423 consensus_queue.cc:260] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
W20250901 01:53:18.195498 1101 leader_election.cc:343] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2ba2450480454388723cd729cc438e (127.0.104.3:42927): Not found: Tablet not found: a2d3b7c130694ac6aa4249091c2c479b
I20250901 01:53:18.199092 1423 ts_tablet_manager.cc:1428] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250901 01:53:18.200767 1424 ts_tablet_manager.cc:1428] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250901 01:53:18.201917 1424 tablet_bootstrap.cc:492] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:18.203758 1424 tablet_bootstrap.cc:654] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.204770 1439 ts_tablet_manager.cc:1918] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:53:18.205001 1439 log.cc:1199] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal/wals/cfb2dd7459a94484b9df36a853de4a57
I20250901 01:53:18.205673 1439 ts_tablet_manager.cc:1939] T cfb2dd7459a94484b9df36a853de4a57 P e3c774a4edf44d6098a2332e734967a5: Deleting consensus metadata
I20250901 01:53:18.207665 1439 tablet_replica.cc:331] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5: stopping tablet replica
I20250901 01:53:18.207883 1439 raft_consensus.cc:2241] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:18.208092 1439 raft_consensus.cc:2270] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:18.209510 673 catalog_manager.cc:4928] TS e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245): tablet cfb2dd7459a94484b9df36a853de4a57 (table default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]) successfully deleted
W20250901 01:53:18.211174 1103 leader_election.cc:343] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Not found: Tablet not found: a2d3b7c130694ac6aa4249091c2c479b
I20250901 01:53:18.211354 1103 leader_election.cc:304] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3c774a4edf44d6098a2332e734967a5; no voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853
W20250901 01:53:18.211766 1103 raft_consensus.cc:2684] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:53:18.212776 1439 ts_tablet_manager.cc:1905] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:18.219574 1439 ts_tablet_manager.cc:1918] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:53:18.219666 681 catalog_manager.cc:2149] created HMS catalog entry for table [id=af5c975082de48cfa7295b2d8bd16635]
I20250901 01:53:18.219869 1439 log.cc:1199] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal/wals/a2d3b7c130694ac6aa4249091c2c479b
I20250901 01:53:18.220232 1439 ts_tablet_manager.cc:1939] T a2d3b7c130694ac6aa4249091c2c479b P e3c774a4edf44d6098a2332e734967a5: Deleting consensus metadata
I20250901 01:53:18.221303 673 catalog_manager.cc:4928] TS e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245): tablet a2d3b7c130694ac6aa4249091c2c479b (table default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647]) successfully deleted
I20250901 01:53:18.228816 1422 tablet_bootstrap.cc:492] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:18.229032 1422 ts_tablet_manager.cc:1397] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.005s
I20250901 01:53:18.229983 1424 tablet_bootstrap.cc:492] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:18.229929 1422 raft_consensus.cc:357] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.230207 1422 raft_consensus.cc:383] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.230295 1422 raft_consensus.cc:738] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.230451 1424 ts_tablet_manager.cc:1397] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.010s
I20250901 01:53:18.231282 1424 raft_consensus.cc:357] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.231547 1424 raft_consensus.cc:383] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.231689 1424 raft_consensus.cc:738] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.232028 1424 consensus_queue.cc:260] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.232975 1422 consensus_queue.cc:260] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.234314 1422 ts_tablet_manager.cc:1428] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250901 01:53:18.234728 1422 tablet_bootstrap.cc:492] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:18.240562 1004 tablet_service.cc:1468] Processing CreateTablet for tablet a6af5a07f3ca4fe4a8808b913551c091 (DEFAULT_TABLE table=default.table_476c1fc1206b4426855b5602d4b0cc88 [id=af5c975082de48cfa7295b2d8bd16635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.241230 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6af5a07f3ca4fe4a8808b913551c091. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.242911 1422 tablet_bootstrap.cc:654] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.244403 1286 tablet_service.cc:1468] Processing CreateTablet for tablet a6af5a07f3ca4fe4a8808b913551c091 (DEFAULT_TABLE table=default.table_476c1fc1206b4426855b5602d4b0cc88 [id=af5c975082de48cfa7295b2d8bd16635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.244869 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6af5a07f3ca4fe4a8808b913551c091. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.245945 1424 ts_tablet_manager.cc:1428] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20250901 01:53:18.251685 1424 tablet_bootstrap.cc:492] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:18.254081 1424 tablet_bootstrap.cc:654] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.255394 1147 tablet_service.cc:1468] Processing CreateTablet for tablet a6af5a07f3ca4fe4a8808b913551c091 (DEFAULT_TABLE table=default.table_476c1fc1206b4426855b5602d4b0cc88 [id=af5c975082de48cfa7295b2d8bd16635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.255921 1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6af5a07f3ca4fe4a8808b913551c091. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.259871 1424 tablet_bootstrap.cc:492] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:18.260030 1424 ts_tablet_manager.cc:1397] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20250901 01:53:18.260643 1424 raft_consensus.cc:357] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.260807 1424 raft_consensus.cc:383] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.260855 1424 raft_consensus.cc:738] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.261101 1424 consensus_queue.cc:260] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.263357 1424 ts_tablet_manager.cc:1428] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:53:18.264391 1422 tablet_bootstrap.cc:492] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:18.264613 1422 ts_tablet_manager.cc:1397] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.007s
I20250901 01:53:18.265730 1422 raft_consensus.cc:357] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.265987 1422 raft_consensus.cc:383] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.266100 1422 raft_consensus.cc:738] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.266454 1422 consensus_queue.cc:260] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.267304 1422 ts_tablet_manager.cc:1428] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250901 01:53:18.267691 1422 tablet_bootstrap.cc:492] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:18.270304 1422 tablet_bootstrap.cc:654] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.278674 1422 tablet_bootstrap.cc:492] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:18.278914 1422 ts_tablet_manager.cc:1397] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20250901 01:53:18.280282 1422 raft_consensus.cc:357] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.280532 1422 raft_consensus.cc:383] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.280622 1422 raft_consensus.cc:738] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.280922 1422 consensus_queue.cc:260] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.281523 1422 ts_tablet_manager.cc:1428] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:53:18.287285 1422 tablet_bootstrap.cc:492] T a6af5a07f3ca4fe4a8808b913551c091 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:18.289032 1424 tablet_bootstrap.cc:492] T a6af5a07f3ca4fe4a8808b913551c091 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:18.289209 1422 tablet_bootstrap.cc:654] T a6af5a07f3ca4fe4a8808b913551c091 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.291044 1424 tablet_bootstrap.cc:654] T a6af5a07f3ca4fe4a8808b913551c091 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.296136 1423 tablet_bootstrap.cc:492] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:18.298558 1423 tablet_bootstrap.cc:654] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.301272 1424 tablet_bootstrap.cc:492] T a6af5a07f3ca4fe4a8808b913551c091 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:18.301491 1424 ts_tablet_manager.cc:1397] T a6af5a07f3ca4fe4a8808b913551c091 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20250901 01:53:18.302686 1424 raft_consensus.cc:357] T a6af5a07f3ca4fe4a8808b913551c091 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:18.302932 1424 raft_consensus.cc:383] T a6af5a07f3ca4fe4a8808b913551c091 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.303022 1424 raft_consensus.cc:738] T a6af5a07f3ca4fe4a8808b913551c091 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.303323 1424 consensus_queue.cc:260] T a6af5a07f3ca4fe4a8808b913551c091 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:18.303941 1424 ts_tablet_manager.cc:1428] T a6af5a07f3ca4fe4a8808b913551c091 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:53:18.306828 1422 tablet_bootstrap.cc:492] T a6af5a07f3ca4fe4a8808b913551c091 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:18.307034 1422 ts_tablet_manager.cc:1397] T a6af5a07f3ca4fe4a8808b913551c091 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.015s
I20250901 01:53:18.307996 1422 raft_consensus.cc:357] T a6af5a07f3ca4fe4a8808b913551c091 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:18.308380 1422 raft_consensus.cc:383] T a6af5a07f3ca4fe4a8808b913551c091 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.308497 1422 raft_consensus.cc:738] T a6af5a07f3ca4fe4a8808b913551c091 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.308768 1422 consensus_queue.cc:260] T a6af5a07f3ca4fe4a8808b913551c091 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:18.310305 683 catalog_manager.cc:2149] created HMS catalog entry for table [id=61c047af6fff411f908946c6206ba414]
I20250901 01:53:18.310626 1422 ts_tablet_manager.cc:1428] T a6af5a07f3ca4fe4a8808b913551c091 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:53:18.316722 1423 tablet_bootstrap.cc:492] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:18.317015 1423 ts_tablet_manager.cc:1397] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.006s
I20250901 01:53:18.318064 1423 raft_consensus.cc:357] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:18.318404 1423 raft_consensus.cc:383] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.318578 1423 raft_consensus.cc:738] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.318876 1423 consensus_queue.cc:260] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:18.318126 683 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_6bcc2bc2c6574eca8387f75fc422beb9" } new_table_name: "default.Table_5d5e7ea4ac7d4e5f9de3237058d11e80" modify_external_catalogs: true
I20250901 01:53:18.319504 1423 ts_tablet_manager.cc:1428] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:53:18.326030 1431 raft_consensus.cc:491] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:18.326354 1431 raft_consensus.cc:513] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.328359 1431 leader_election.cc:290] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:18.336977 1147 tablet_service.cc:1468] Processing CreateTablet for tablet 8d18463c27854cd28fa9de30c4990749 (DEFAULT_TABLE table=default.table_795876dee30e438da68abc8a5bbae13f [id=61c047af6fff411f908946c6206ba414]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.337589 1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d18463c27854cd28fa9de30c4990749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.343235 1004 tablet_service.cc:1468] Processing CreateTablet for tablet 8d18463c27854cd28fa9de30c4990749 (DEFAULT_TABLE table=default.table_795876dee30e438da68abc8a5bbae13f [id=61c047af6fff411f908946c6206ba414]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.343756 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d18463c27854cd28fa9de30c4990749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.346508 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6380a3fefba4c4caed049b434425733" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:18.346932 1024 raft_consensus.cc:2466] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:18.347568 1243 leader_election.cc:304] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters:
I20250901 01:53:18.348070 1431 raft_consensus.cc:2802] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:18.348238 1431 raft_consensus.cc:491] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:18.348357 1431 raft_consensus.cc:3058] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.342638 1286 tablet_service.cc:1468] Processing CreateTablet for tablet 8d18463c27854cd28fa9de30c4990749 (DEFAULT_TABLE table=default.table_795876dee30e438da68abc8a5bbae13f [id=61c047af6fff411f908946c6206ba414]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.350643 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d18463c27854cd28fa9de30c4990749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.353904 681 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_e628b8e1d2214e458c8a283659d45a07" } new_table_name: "default.Table_7e006da3e6d24d309af545afeac11ca4" modify_external_catalogs: true
I20250901 01:53:18.351455 1431 raft_consensus.cc:513] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.360783 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6380a3fefba4c4caed049b434425733" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:18.361039 1024 raft_consensus.cc:3058] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.362331 1424 tablet_bootstrap.cc:492] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:18.362602 1422 tablet_bootstrap.cc:492] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:18.363976 1024 raft_consensus.cc:2466] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:18.364395 1422 tablet_bootstrap.cc:654] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.364411 1424 tablet_bootstrap.cc:654] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.367014 1423 tablet_bootstrap.cc:492] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:18.367364 1422 tablet_bootstrap.cc:492] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:18.367650 1422 ts_tablet_manager.cc:1397] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20250901 01:53:18.368775 1422 raft_consensus.cc:357] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.369145 1422 raft_consensus.cc:383] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.369302 1423 tablet_bootstrap.cc:654] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.369294 1422 raft_consensus.cc:738] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.369684 1422 consensus_queue.cc:260] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.371316 1422 ts_tablet_manager.cc:1428] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:53:18.373245 1424 tablet_bootstrap.cc:492] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:18.373482 1424 ts_tablet_manager.cc:1397] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20250901 01:53:18.374442 1424 raft_consensus.cc:357] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.374672 1424 raft_consensus.cc:383] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.374751 1424 raft_consensus.cc:738] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.374996 1424 consensus_queue.cc:260] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.379129 1431 leader_election.cc:290] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Requested vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:18.379434 1431 raft_consensus.cc:491] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:18.379608 1431 raft_consensus.cc:513] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.380234 1431 leader_election.cc:290] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:18.383942 1423 tablet_bootstrap.cc:492] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:18.384256 1423 ts_tablet_manager.cc:1397] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250901 01:53:18.385378 1423 raft_consensus.cc:357] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.385679 1423 raft_consensus.cc:383] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.385776 1423 raft_consensus.cc:738] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.386101 1423 consensus_queue.cc:260] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.386876 1423 ts_tablet_manager.cc:1428] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250901 01:53:18.395350 1424 ts_tablet_manager.cc:1428] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.022s user 0.005s sys 0.005s
I20250901 01:53:18.401743 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6380a3fefba4c4caed049b434425733" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:18.402043 1167 raft_consensus.cc:2466] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:18.402882 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6380a3fefba4c4caed049b434425733" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:18.403100 1167 raft_consensus.cc:3058] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.406351 1167 raft_consensus.cc:2466] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:18.407284 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3c5c349f8f8435f8b125a6d527f970a" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:18.407581 1167 raft_consensus.cc:2466] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:18.408622 1240 leader_election.cc:304] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:18.408987 1240 leader_election.cc:304] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:18.409777 684 catalog_manager.cc:2149] created HMS catalog entry for table [id=31d721fb8b204568a0190dda6def01cd]
I20250901 01:53:18.410120 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3c5c349f8f8435f8b125a6d527f970a" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:18.410432 1024 raft_consensus.cc:2466] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:18.412194 1467 raft_consensus.cc:2802] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:18.412331 1467 raft_consensus.cc:491] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:18.412493 1467 raft_consensus.cc:3058] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.409231 1463 raft_consensus.cc:2802] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:18.413627 1463 raft_consensus.cc:695] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [term 1 LEADER]: Becoming Leader. State: Replica: ad2ba2450480454388723cd729cc438e, State: Running, Role: LEADER
I20250901 01:53:18.414243 1463 consensus_queue.cc:237] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.416085 1467 raft_consensus.cc:513] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.416800 1467 leader_election.cc:290] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Requested vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:18.417451 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3c5c349f8f8435f8b125a6d527f970a" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:18.417678 1024 raft_consensus.cc:3058] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.421546 1024 raft_consensus.cc:2466] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:18.422441 1243 leader_election.cc:304] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters:
I20250901 01:53:18.422866 1467 raft_consensus.cc:2802] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:18.423059 1467 raft_consensus.cc:695] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [term 1 LEADER]: Becoming Leader. State: Replica: ad2ba2450480454388723cd729cc438e, State: Running, Role: LEADER
I20250901 01:53:18.423332 1467 consensus_queue.cc:237] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.423476 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3c5c349f8f8435f8b125a6d527f970a" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:18.423676 1167 raft_consensus.cc:3058] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.427448 1167 raft_consensus.cc:2466] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:18.435469 1286 tablet_service.cc:1468] Processing CreateTablet for tablet b56b8dbd4e5b4892aa3b50409fc4584c (DEFAULT_TABLE table=default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.435967 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b56b8dbd4e5b4892aa3b50409fc4584c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.437844 684 master_service.cc:948] ReplaceTablet: received request to replace tablet a3c5c349f8f8435f8b125a6d527f970a from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:18.440440 1004 tablet_service.cc:1468] Processing CreateTablet for tablet b56b8dbd4e5b4892aa3b50409fc4584c (DEFAULT_TABLE table=default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.440915 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b56b8dbd4e5b4892aa3b50409fc4584c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.445331 1424 tablet_bootstrap.cc:492] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:18.447345 1424 tablet_bootstrap.cc:654] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.453233 1147 tablet_service.cc:1468] Processing CreateTablet for tablet b56b8dbd4e5b4892aa3b50409fc4584c (DEFAULT_TABLE table=default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.453764 1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b56b8dbd4e5b4892aa3b50409fc4584c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.457072 684 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Sending DeleteTablet for 3 replicas of tablet a3c5c349f8f8435f8b125a6d527f970a
I20250901 01:53:18.457938 1003 tablet_service.cc:1515] Processing DeleteTablet for tablet a3c5c349f8f8435f8b125a6d527f970a with delete_type TABLET_DATA_DELETED (replaced by tablet 586fbc1ef054435aaaea9ae961f15115) from {username='slave'} at 127.0.0.1:35622
I20250901 01:53:18.458303 1438 tablet_replica.cc:331] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853: stopping tablet replica
I20250901 01:53:18.458545 1438 raft_consensus.cc:2241] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:18.458750 1438 raft_consensus.cc:2270] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:18.458802 1146 tablet_service.cc:1515] Processing DeleteTablet for tablet a3c5c349f8f8435f8b125a6d527f970a with delete_type TABLET_DATA_DELETED (replaced by tablet 586fbc1ef054435aaaea9ae961f15115) from {username='slave'} at 127.0.0.1:53914
I20250901 01:53:18.459082 1439 tablet_replica.cc:331] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5: stopping tablet replica
I20250901 01:53:18.459285 1439 raft_consensus.cc:2241] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:18.459417 684 catalog_manager.cc:6652] ReplaceTablet: tablet a3c5c349f8f8435f8b125a6d527f970a deleted and replaced by tablet 586fbc1ef054435aaaea9ae961f15115
I20250901 01:53:18.459481 1439 raft_consensus.cc:2270] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:18.459592 1438 ts_tablet_manager.cc:1905] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:18.460381 1286 tablet_service.cc:1515] Processing DeleteTablet for tablet a3c5c349f8f8435f8b125a6d527f970a with delete_type TABLET_DATA_DELETED (replaced by tablet 586fbc1ef054435aaaea9ae961f15115) from {username='slave'} at 127.0.0.1:58732
I20250901 01:53:18.460668 1440 tablet_replica.cc:331] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e: stopping tablet replica
I20250901 01:53:18.460647 1439 ts_tablet_manager.cc:1905] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:18.463879 1422 tablet_bootstrap.cc:492] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:18.465917 1422 tablet_bootstrap.cc:654] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.466555 1438 ts_tablet_manager.cc:1918] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:53:18.466718 1438 log.cc:1199] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/wals/a3c5c349f8f8435f8b125a6d527f970a
I20250901 01:53:18.467094 1438 ts_tablet_manager.cc:1939] T a3c5c349f8f8435f8b125a6d527f970a P b65b3154aefd49f7a37f70e9207e9853: Deleting consensus metadata
I20250901 01:53:18.468452 676 catalog_manager.cc:4928] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): tablet a3c5c349f8f8435f8b125a6d527f970a (table default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647]) successfully deleted
I20250901 01:53:18.472510 1424 tablet_bootstrap.cc:492] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:18.472713 1424 ts_tablet_manager.cc:1397] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.007s
I20250901 01:53:18.473635 1424 raft_consensus.cc:357] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.473850 1424 raft_consensus.cc:383] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.473970 1424 raft_consensus.cc:738] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.474315 1424 consensus_queue.cc:260] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.474880 1424 ts_tablet_manager.cc:1428] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:53:18.480649 1422 tablet_bootstrap.cc:492] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:18.480937 1422 ts_tablet_manager.cc:1397] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20250901 01:53:18.481972 1422 raft_consensus.cc:357] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.482095 1003 tablet_service.cc:1468] Processing CreateTablet for tablet 586fbc1ef054435aaaea9ae961f15115 (DEFAULT_TABLE table=default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.482371 1422 raft_consensus.cc:383] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.482553 1422 raft_consensus.cc:738] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.482616 1003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 586fbc1ef054435aaaea9ae961f15115. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.482910 1422 consensus_queue.cc:260] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.487303 1422 ts_tablet_manager.cc:1428] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250901 01:53:18.491236 1423 tablet_bootstrap.cc:492] T 586fbc1ef054435aaaea9ae961f15115 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:18.493239 1147 tablet_service.cc:1468] Processing CreateTablet for tablet 586fbc1ef054435aaaea9ae961f15115 (DEFAULT_TABLE table=default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.493299 1286 tablet_service.cc:1468] Processing CreateTablet for tablet 586fbc1ef054435aaaea9ae961f15115 (DEFAULT_TABLE table=default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.493466 1423 tablet_bootstrap.cc:654] T 586fbc1ef054435aaaea9ae961f15115 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.493747 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 586fbc1ef054435aaaea9ae961f15115. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.493862 1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 586fbc1ef054435aaaea9ae961f15115. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.498656 686 catalog_manager.cc:2149] created HMS catalog entry for table [id=ed41948c8dd04b93b7d5fc44cede27f6]
I20250901 01:53:18.500942 1422 tablet_bootstrap.cc:492] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:18.502930 1423 tablet_bootstrap.cc:492] T 586fbc1ef054435aaaea9ae961f15115 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:18.503014 1422 tablet_bootstrap.cc:654] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.503152 1423 ts_tablet_manager.cc:1397] T 586fbc1ef054435aaaea9ae961f15115 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20250901 01:53:18.503997 1423 raft_consensus.cc:357] T 586fbc1ef054435aaaea9ae961f15115 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.504237 1423 raft_consensus.cc:383] T 586fbc1ef054435aaaea9ae961f15115 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.504359 1423 raft_consensus.cc:738] T 586fbc1ef054435aaaea9ae961f15115 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.504657 1423 consensus_queue.cc:260] T 586fbc1ef054435aaaea9ae961f15115 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.505261 1423 ts_tablet_manager.cc:1428] T 586fbc1ef054435aaaea9ae961f15115 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:53:18.505656 1423 tablet_bootstrap.cc:492] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:18.506218 686 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58182 (request call id 21) took 1005 ms. Trace:
I20250901 01:53:18.507655 1423 tablet_bootstrap.cc:654] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.506413 686 rpcz_store.cc:276] 0901 01:53:17.500600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:17.502124 (+ 1524us) service_pool.cc:225] Handling call
0901 01:53:17.587442 (+ 85318us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:53:17.587882 (+ 440us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:53:18.498579 (+910697us) catalog_manager.cc:2148] Created new table in HMS catalog
0901 01:53:18.505890 (+ 7311us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:53:18.506030 (+ 140us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:53:18.506142 (+ 112us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:53:18.499777 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:53:18.499926 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:53:18.499933 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:53:18.499936 (+ 3us) tablet.cc:660] Decoding operations
0901 01:53:18.500046 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:53:18.500063 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0901 01:53:18.500069 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:53:18.500155 (+ 86us) tablet.cc:699] Row locks acquired
0901 01:53:18.500157 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:53:18.500207 (+ 50us) write_op.cc:270] Start()
0901 01:53:18.500249 (+ 42us) write_op.cc:276] Timestamp: P: 1756691598500184 usec, L: 0
0901 01:53:18.500253 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:53:18.500505 (+ 252us) log.cc:844] Serialized 432 byte log entry
0901 01:53:18.505177 (+ 4672us) op_driver.cc:464] REPLICATION: finished
0901 01:53:18.505347 (+ 170us) write_op.cc:301] APPLY: starting
0901 01:53:18.505364 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0901 01:53:18.505414 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0901 01:53:18.505449 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:53:18.505557 (+ 108us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:53:18.505594 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:53:18.505601 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:53:18.505687 (+ 86us) log.cc:844] Serialized 28 byte log entry
0901 01:53:18.505759 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:18.505782 (+ 23us) write_op.cc:454] Released schema lock
0901 01:53:18.505789 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0901 01:53:18.505879 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:18.505882 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":821715,"Hive Metastore.run_cpu_time_us":344,"Hive Metastore.run_wall_time_us":88475,"child_traces":[["op",{"apply.queue_time_us":115,"apply.run_cpu_time_us":518,"apply.run_wall_time_us":517,"num_ops":2,"prepare.queue_time_us":92,"prepare.run_cpu_time_us":873,"prepare.run_wall_time_us":1220,"raft.queue_time_us":112,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":266,"replication_time_us":4889}]]}
I20250901 01:53:18.510478 1439 ts_tablet_manager.cc:1918] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:53:18.510615 1439 log.cc:1199] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal/wals/a3c5c349f8f8435f8b125a6d527f970a
I20250901 01:53:18.510989 1439 ts_tablet_manager.cc:1939] T a3c5c349f8f8435f8b125a6d527f970a P e3c774a4edf44d6098a2332e734967a5: Deleting consensus metadata
I20250901 01:53:18.512006 684 master_service.cc:948] ReplaceTablet: received request to replace tablet 6263cf06150b49fbbda6c89bd1fb979e from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:18.512115 673 catalog_manager.cc:4928] TS e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245): tablet a3c5c349f8f8435f8b125a6d527f970a (table default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647]) successfully deleted
I20250901 01:53:18.513569 1424 tablet_bootstrap.cc:492] T 586fbc1ef054435aaaea9ae961f15115 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:18.515856 1424 tablet_bootstrap.cc:654] T 586fbc1ef054435aaaea9ae961f15115 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.517298 686 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_476c1fc1206b4426855b5602d4b0cc88" } new_table_name: "default.Table_1ff7214e853748e58232b9bea0cb6912" modify_external_catalogs: true
I20250901 01:53:18.519850 1423 tablet_bootstrap.cc:492] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:18.520202 1423 ts_tablet_manager.cc:1397] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20250901 01:53:18.521070 1423 raft_consensus.cc:357] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.521315 1423 raft_consensus.cc:383] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.521405 1423 raft_consensus.cc:738] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.522848 1423 consensus_queue.cc:260] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.527493 1423 ts_tablet_manager.cc:1428] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250901 01:53:18.528044 1463 raft_consensus.cc:491] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:18.528211 684 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Sending DeleteTablet for 3 replicas of tablet 6263cf06150b49fbbda6c89bd1fb979e
I20250901 01:53:18.528251 1463 raft_consensus.cc:513] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.528985 1463 leader_election.cc:290] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:18.529408 1424 tablet_bootstrap.cc:492] T 586fbc1ef054435aaaea9ae961f15115 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:18.529495 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cd7993f6a99468caa9b13fc4e8c0ba1" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:18.529593 1424 ts_tablet_manager.cc:1397] T 586fbc1ef054435aaaea9ae961f15115 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20250901 01:53:18.529623 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cd7993f6a99468caa9b13fc4e8c0ba1" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:18.529738 1167 raft_consensus.cc:2466] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:18.529819 1024 raft_consensus.cc:2466] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:18.530277 1240 leader_election.cc:304] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:18.530767 1463 raft_consensus.cc:2802] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:18.530650 1424 raft_consensus.cc:357] T 586fbc1ef054435aaaea9ae961f15115 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.530866 1463 raft_consensus.cc:491] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:18.530925 1424 raft_consensus.cc:383] T 586fbc1ef054435aaaea9ae961f15115 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.530983 1463 raft_consensus.cc:3058] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.531024 1424 raft_consensus.cc:738] T 586fbc1ef054435aaaea9ae961f15115 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.531390 1424 consensus_queue.cc:260] T 586fbc1ef054435aaaea9ae961f15115 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.532339 1424 ts_tablet_manager.cc:1428] T 586fbc1ef054435aaaea9ae961f15115 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250901 01:53:18.533986 1463 raft_consensus.cc:513] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.535964 1463 leader_election.cc:290] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Requested vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:18.536150 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cd7993f6a99468caa9b13fc4e8c0ba1" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:18.536374 1024 raft_consensus.cc:3058] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.536827 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cd7993f6a99468caa9b13fc4e8c0ba1" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:18.537052 1167 raft_consensus.cc:3058] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.539551 1024 raft_consensus.cc:2466] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:18.539979 1167 raft_consensus.cc:2466] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:18.540853 1243 leader_election.cc:304] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:18.540939 1004 tablet_service.cc:1515] Processing DeleteTablet for tablet 6263cf06150b49fbbda6c89bd1fb979e with delete_type TABLET_DATA_DELETED (replaced by tablet 138d19e14b734c1cace0d7ba51ab2caa) from {username='slave'} at 127.0.0.1:35622
I20250901 01:53:18.541523 1463 raft_consensus.cc:2802] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:18.541781 1463 raft_consensus.cc:695] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [term 1 LEADER]: Becoming Leader. State: Replica: ad2ba2450480454388723cd729cc438e, State: Running, Role: LEADER
I20250901 01:53:18.541853 1438 tablet_replica.cc:331] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853: stopping tablet replica
I20250901 01:53:18.542042 1438 raft_consensus.cc:2241] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:18.542264 1438 raft_consensus.cc:2270] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:18.542181 1463 consensus_queue.cc:237] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.543344 1438 ts_tablet_manager.cc:1905] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:18.551817 1422 tablet_bootstrap.cc:492] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:18.552035 1422 ts_tablet_manager.cc:1397] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.005s
I20250901 01:53:18.553043 1147 tablet_service.cc:1515] Processing DeleteTablet for tablet 6263cf06150b49fbbda6c89bd1fb979e with delete_type TABLET_DATA_DELETED (replaced by tablet 138d19e14b734c1cace0d7ba51ab2caa) from {username='slave'} at 127.0.0.1:53914
I20250901 01:53:18.553380 1422 raft_consensus.cc:357] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.560737 1422 raft_consensus.cc:383] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.560885 1422 raft_consensus.cc:738] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.562225 1422 consensus_queue.cc:260] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.567745 1422 ts_tablet_manager.cc:1428] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20250901 01:53:18.572062 1438 ts_tablet_manager.cc:1918] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:53:18.572361 1438 log.cc:1199] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/wals/6263cf06150b49fbbda6c89bd1fb979e
I20250901 01:53:18.572963 1438 ts_tablet_manager.cc:1939] T 6263cf06150b49fbbda6c89bd1fb979e P b65b3154aefd49f7a37f70e9207e9853: Deleting consensus metadata
I20250901 01:53:18.574453 676 catalog_manager.cc:4928] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): tablet 6263cf06150b49fbbda6c89bd1fb979e (table default.table_6bcc2bc2c6574eca8387f75fc422beb9 [id=8ee4e0d7bca3410b9967df0dacf8614d]) successfully deleted
I20250901 01:53:18.575287 1286 tablet_service.cc:1515] Processing DeleteTablet for tablet 6263cf06150b49fbbda6c89bd1fb979e with delete_type TABLET_DATA_DELETED (replaced by tablet 138d19e14b734c1cace0d7ba51ab2caa) from {username='slave'} at 127.0.0.1:58732
I20250901 01:53:18.575731 684 catalog_manager.cc:6652] ReplaceTablet: tablet 6263cf06150b49fbbda6c89bd1fb979e deleted and replaced by tablet 138d19e14b734c1cace0d7ba51ab2caa
I20250901 01:53:18.576030 1439 tablet_replica.cc:331] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5: stopping tablet replica
I20250901 01:53:18.577996 1286 tablet_service.cc:1468] Processing CreateTablet for tablet a774cb6ccef24d958ba6c898c5e4786c (DEFAULT_TABLE table=default.table_832e2227a6314ac2a050f76013e116fc [id=ed41948c8dd04b93b7d5fc44cede27f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.578801 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a774cb6ccef24d958ba6c898c5e4786c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.587458 1450 raft_consensus.cc:491] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:18.587668 1450 raft_consensus.cc:513] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.588533 1450 leader_election.cc:290] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:18.590122 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "586fbc1ef054435aaaea9ae961f15115" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:18.590490 1307 raft_consensus.cc:2466] T 586fbc1ef054435aaaea9ae961f15115 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 0.
I20250901 01:53:18.591318 1101 leader_election.cc:304] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:18.591820 1450 raft_consensus.cc:2802] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:18.591955 1450 raft_consensus.cc:491] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:18.592034 1450 raft_consensus.cc:3058] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.595012 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "586fbc1ef054435aaaea9ae961f15115" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:18.595283 1024 raft_consensus.cc:2466] T 586fbc1ef054435aaaea9ae961f15115 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 0.
I20250901 01:53:18.595413 1440 raft_consensus.cc:2241] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:53:18.596488 1440 pending_rounds.cc:70] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e: Trying to abort 1 pending ops.
I20250901 01:53:18.596709 1440 pending_rounds.cc:77] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408787145084928 op_type: NO_OP noop_request { }
I20250901 01:53:18.596957 1440 raft_consensus.cc:2887] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:53:18.597136 1440 raft_consensus.cc:2270] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:18.597559 1450 raft_consensus.cc:513] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.599646 1440 ts_tablet_manager.cc:1905] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:18.600531 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "586fbc1ef054435aaaea9ae961f15115" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:18.600762 1024 raft_consensus.cc:3058] T 586fbc1ef054435aaaea9ae961f15115 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.602378 1434 raft_consensus.cc:491] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:18.602556 1434 raft_consensus.cc:513] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.603250 1434 leader_election.cc:290] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:18.604091 1306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d18463c27854cd28fa9de30c4990749" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:18.604483 1306 raft_consensus.cc:2466] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:18.604975 954 leader_election.cc:304] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters:
I20250901 01:53:18.605402 1024 raft_consensus.cc:2466] T 586fbc1ef054435aaaea9ae961f15115 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 1.
I20250901 01:53:18.605782 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "586fbc1ef054435aaaea9ae961f15115" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:18.605952 1307 raft_consensus.cc:3058] T 586fbc1ef054435aaaea9ae961f15115 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.609066 1307 raft_consensus.cc:2466] T 586fbc1ef054435aaaea9ae961f15115 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 1.
I20250901 01:53:18.609222 1440 ts_tablet_manager.cc:1918] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:18.609350 1440 log.cc:1199] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal/wals/a3c5c349f8f8435f8b125a6d527f970a
I20250901 01:53:18.609869 1434 raft_consensus.cc:2802] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:18.609966 1440 ts_tablet_manager.cc:1939] T a3c5c349f8f8435f8b125a6d527f970a P ad2ba2450480454388723cd729cc438e: Deleting consensus metadata
I20250901 01:53:18.609978 1434 raft_consensus.cc:491] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:18.610098 1434 raft_consensus.cc:3058] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.611300 674 catalog_manager.cc:4928] TS ad2ba2450480454388723cd729cc438e (127.0.104.3:42927): tablet a3c5c349f8f8435f8b125a6d527f970a (table default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647]) successfully deleted
I20250901 01:53:18.611791 1440 tablet_replica.cc:331] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e: stopping tablet replica
I20250901 01:53:18.611990 1440 raft_consensus.cc:2241] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:18.612260 1440 raft_consensus.cc:2270] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:18.613015 1440 ts_tablet_manager.cc:1905] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:18.613440 1103 leader_election.cc:304] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:18.613778 1429 raft_consensus.cc:2802] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:18.619611 1434 raft_consensus.cc:513] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.621251 1166 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d18463c27854cd28fa9de30c4990749" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:18.621549 1166 raft_consensus.cc:3058] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.624492 685 catalog_manager.cc:2149] created HMS catalog entry for table [id=2e950743029d48a49f05207cd5faee19]
I20250901 01:53:18.625540 1166 raft_consensus.cc:2466] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:18.627487 953 leader_election.cc:304] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:18.628156 1478 raft_consensus.cc:2802] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:18.628178 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d18463c27854cd28fa9de30c4990749" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:18.628407 1167 raft_consensus.cc:2374] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1. Re-sending same reply.
I20250901 01:53:18.631008 1478 raft_consensus.cc:695] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Becoming Leader. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Running, Role: LEADER
W20250901 01:53:18.631070 676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20250901 01:53:18.631392 1478 consensus_queue.cc:237] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.635406 1479 raft_consensus.cc:491] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:18.635892 1479 raft_consensus.cc:513] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.636634 1434 leader_election.cc:290] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Requested vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:18.636960 1450 leader_election.cc:290] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Requested vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:18.637480 1306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d18463c27854cd28fa9de30c4990749" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:18.637728 1306 raft_consensus.cc:3058] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.641660 1306 raft_consensus.cc:2466] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:18.642882 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b56b8dbd4e5b4892aa3b50409fc4584c" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:18.643050 1440 ts_tablet_manager.cc:1918] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:53:18.643168 1307 raft_consensus.cc:2466] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:18.638239 1429 raft_consensus.cc:695] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [term 1 LEADER]: Becoming Leader. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Running, Role: LEADER
I20250901 01:53:18.642705 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b56b8dbd4e5b4892aa3b50409fc4584c" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:18.644423 1167 raft_consensus.cc:2466] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:18.645514 1429 consensus_queue.cc:237] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.641342 1479 leader_election.cc:290] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2ba2450480454388723cd729cc438e (127.0.104.3:42927), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:18.643720 954 leader_election.cc:304] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters:
I20250901 01:53:18.649806 1479 raft_consensus.cc:2802] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:18.649959 1479 raft_consensus.cc:491] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:18.650048 1479 raft_consensus.cc:3058] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.643186 1440 log.cc:1199] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal/wals/6263cf06150b49fbbda6c89bd1fb979e
I20250901 01:53:18.652717 1440 ts_tablet_manager.cc:1939] T 6263cf06150b49fbbda6c89bd1fb979e P ad2ba2450480454388723cd729cc438e: Deleting consensus metadata
I20250901 01:53:18.654067 685 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58182 (request call id 22) took 1153 ms. Trace:
I20250901 01:53:18.654311 685 rpcz_store.cc:276] 0901 01:53:17.500765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:17.502203 (+ 1438us) service_pool.cc:225] Handling call
0901 01:53:17.588322 (+ 86119us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:53:17.588743 (+ 421us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:53:18.624420 (+1035677us) catalog_manager.cc:2148] Created new table in HMS catalog
0901 01:53:18.653739 (+ 29319us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:53:18.653897 (+ 158us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:53:18.654010 (+ 113us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:53:18.632491 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:53:18.632646 (+ 155us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:53:18.632653 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:53:18.632655 (+ 2us) tablet.cc:660] Decoding operations
0901 01:53:18.635109 (+ 2454us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:53:18.635125 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0901 01:53:18.635131 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:53:18.635231 (+ 100us) tablet.cc:699] Row locks acquired
0901 01:53:18.635233 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:53:18.635276 (+ 43us) write_op.cc:270] Start()
0901 01:53:18.635317 (+ 41us) write_op.cc:276] Timestamp: P: 1756691598635259 usec, L: 0
0901 01:53:18.635321 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:53:18.635567 (+ 246us) log.cc:844] Serialized 432 byte log entry
0901 01:53:18.652841 (+ 17274us) op_driver.cc:464] REPLICATION: finished
0901 01:53:18.653061 (+ 220us) write_op.cc:301] APPLY: starting
0901 01:53:18.653091 (+ 30us) tablet.cc:1365] starting BulkCheckPresence
0901 01:53:18.653146 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0901 01:53:18.653232 (+ 86us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:53:18.653390 (+ 158us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:53:18.653430 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:53:18.653438 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:53:18.653533 (+ 95us) log.cc:844] Serialized 28 byte log entry
0901 01:53:18.653605 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:18.653628 (+ 23us) write_op.cc:454] Released schema lock
0901 01:53:18.653635 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0901 01:53:18.653719 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:18.653723 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":909389,"Hive Metastore.run_cpu_time_us":380,"Hive Metastore.run_wall_time_us":126083,"spinlock_wait_cycles":33024,"child_traces":[["op",{"apply.queue_time_us":154,"apply.run_cpu_time_us":655,"apply.run_wall_time_us":654,"num_ops":2,"prepare.queue_time_us":7131,"prepare.run_cpu_time_us":968,"prepare.run_wall_time_us":13554,"raft.queue_time_us":18740,"raft.run_cpu_time_us":341,"raft.run_wall_time_us":341,"replication_time_us":17484}]]}
I20250901 01:53:18.654984 1439 raft_consensus.cc:2241] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:18.655174 1147 tablet_service.cc:1468] Processing CreateTablet for tablet a774cb6ccef24d958ba6c898c5e4786c (DEFAULT_TABLE table=default.table_832e2227a6314ac2a050f76013e116fc [id=ed41948c8dd04b93b7d5fc44cede27f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.655665 1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a774cb6ccef24d958ba6c898c5e4786c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.657848 685 catalog_manager.cc:5582] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e reported cstate change: term changed from 0 to 1, leader changed from <none> to ad2ba2450480454388723cd729cc438e (127.0.104.3). New cstate: current_term: 1 leader_uuid: "ad2ba2450480454388723cd729cc438e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } health_report { overall_health: HEALTHY } } }
I20250901 01:53:18.661247 1479 raft_consensus.cc:513] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.664878 1424 tablet_bootstrap.cc:492] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:18.665750 684 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_da487bf0d57f439bb9da2ec57880a73b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:18.666658 674 catalog_manager.cc:4928] TS ad2ba2450480454388723cd729cc438e (127.0.104.3:42927): tablet 6263cf06150b49fbbda6c89bd1fb979e (table default.table_6bcc2bc2c6574eca8387f75fc422beb9 [id=8ee4e0d7bca3410b9967df0dacf8614d]) successfully deleted
I20250901 01:53:18.666867 1424 tablet_bootstrap.cc:654] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
W20250901 01:53:18.668972 1428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:53:18.669546 1428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:53:18.669752 1428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250901 01:53:18.670554 1422 tablet_bootstrap.cc:492] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:18.670951 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b56b8dbd4e5b4892aa3b50409fc4584c" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:18.671149 1167 raft_consensus.cc:3058] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.672544 1422 tablet_bootstrap.cc:654] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.674444 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b56b8dbd4e5b4892aa3b50409fc4584c" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:18.674685 1307 raft_consensus.cc:3058] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.674898 1167 raft_consensus.cc:2466] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:18.675519 953 leader_election.cc:304] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:18.675899 1478 raft_consensus.cc:2802] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:18.676492 1478 raft_consensus.cc:695] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Becoming Leader. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Running, Role: LEADER
I20250901 01:53:18.676779 1478 consensus_queue.cc:237] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.677477 1479 leader_election.cc:290] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Requested vote from peers ad2ba2450480454388723cd729cc438e (127.0.104.3:42927), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:18.687469 1307 raft_consensus.cc:2466] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:18.688751 1004 tablet_service.cc:1468] Processing CreateTablet for tablet a774cb6ccef24d958ba6c898c5e4786c (DEFAULT_TABLE table=default.table_832e2227a6314ac2a050f76013e116fc [id=ed41948c8dd04b93b7d5fc44cede27f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.689172 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a774cb6ccef24d958ba6c898c5e4786c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.693924 1423 tablet_bootstrap.cc:492] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:18.695922 1423 tablet_bootstrap.cc:654] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.700012 1478 raft_consensus.cc:491] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:18.700521 1478 raft_consensus.cc:513] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:18.702023 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a6af5a07f3ca4fe4a8808b913551c091" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:18.702369 1167 raft_consensus.cc:2466] T a6af5a07f3ca4fe4a8808b913551c091 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:18.702838 953 leader_election.cc:304] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:18.703361 1479 raft_consensus.cc:2802] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:18.703523 1479 raft_consensus.cc:491] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:18.703702 1479 raft_consensus.cc:3058] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.707542 1478 leader_election.cc:290] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:18.705364 1286 tablet_service.cc:1515] Processing DeleteTablet for tablet a3c5c349f8f8435f8b125a6d527f970a with delete_type TABLET_DATA_DELETED (replaced by tablet 586fbc1ef054435aaaea9ae961f15115) from {username='slave'} at 127.0.0.1:58732
I20250901 01:53:18.707332 1306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a6af5a07f3ca4fe4a8808b913551c091" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:18.711217 1306 raft_consensus.cc:2466] T a6af5a07f3ca4fe4a8808b913551c091 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
W20250901 01:53:18.711494 674 catalog_manager.cc:4895] TS ad2ba2450480454388723cd729cc438e (127.0.104.3:42927): delete failed for tablet a3c5c349f8f8435f8b125a6d527f970a because the tablet was not found. No further retry: Not found: Tablet not found: a3c5c349f8f8435f8b125a6d527f970a
I20250901 01:53:18.655692 1439 raft_consensus.cc:2270] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:53:18.712006 1428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 01:53:18.712145 1428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:53:18.712252 1428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 01:53:18.712329 1428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 01:53:18.709316 1479 raft_consensus.cc:513] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:18.716004 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a6af5a07f3ca4fe4a8808b913551c091" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:18.716279 1167 raft_consensus.cc:3058] T a6af5a07f3ca4fe4a8808b913551c091 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.719655 1167 raft_consensus.cc:2466] T a6af5a07f3ca4fe4a8808b913551c091 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:18.719506 685 catalog_manager.cc:5582] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 reported cstate change: term changed from 0 to 1, leader changed from <none> to b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b65b3154aefd49f7a37f70e9207e9853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } health_report { overall_health: UNKNOWN } } }
I20250901 01:53:18.720633 953 leader_election.cc:304] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:18.721004 1286 tablet_service.cc:1468] Processing CreateTablet for tablet 138d19e14b734c1cace0d7ba51ab2caa (DEFAULT_TABLE table=default.table_6bcc2bc2c6574eca8387f75fc422beb9 [id=8ee4e0d7bca3410b9967df0dacf8614d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.721140 1478 raft_consensus.cc:2802] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:18.721575 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 138d19e14b734c1cace0d7ba51ab2caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.723126 1478 raft_consensus.cc:695] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Becoming Leader. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Running, Role: LEADER
I20250901 01:53:18.723507 1478 consensus_queue.cc:237] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:18.729285 1004 tablet_service.cc:1468] Processing CreateTablet for tablet 138d19e14b734c1cace0d7ba51ab2caa (DEFAULT_TABLE table=default.table_6bcc2bc2c6574eca8387f75fc422beb9 [id=8ee4e0d7bca3410b9967df0dacf8614d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.731467 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 138d19e14b734c1cace0d7ba51ab2caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.731840 1147 tablet_service.cc:1468] Processing CreateTablet for tablet 138d19e14b734c1cace0d7ba51ab2caa (DEFAULT_TABLE table=default.table_6bcc2bc2c6574eca8387f75fc422beb9 [id=8ee4e0d7bca3410b9967df0dacf8614d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.732409 1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 138d19e14b734c1cace0d7ba51ab2caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.730477 1479 leader_election.cc:290] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Requested vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:18.732266 1423 tablet_bootstrap.cc:492] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:18.734246 1423 ts_tablet_manager.cc:1397] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.003s
I20250901 01:53:18.735183 1423 raft_consensus.cc:357] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.735404 1423 raft_consensus.cc:383] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.735491 1423 raft_consensus.cc:738] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.735778 1423 consensus_queue.cc:260] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.736315 1423 ts_tablet_manager.cc:1428] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:53:18.736596 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a6af5a07f3ca4fe4a8808b913551c091" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:18.737334 1439 ts_tablet_manager.cc:1905] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:18.737782 1307 raft_consensus.cc:3058] T a6af5a07f3ca4fe4a8808b913551c091 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.741233 1307 raft_consensus.cc:2466] T a6af5a07f3ca4fe4a8808b913551c091 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:18.749749 1423 tablet_bootstrap.cc:492] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:18.751281 1428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:36985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=35767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
I20250901 01:53:18.751652 1423 tablet_bootstrap.cc:654] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.754839 1428 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:53:18.757614 1428 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250901 01:53:18.761880 1423 tablet_bootstrap.cc:492] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:18.762096 1423 ts_tablet_manager.cc:1397] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20250901 01:53:18.763001 1423 raft_consensus.cc:357] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.763257 1423 raft_consensus.cc:383] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.763345 1423 raft_consensus.cc:738] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.763700 1423 consensus_queue.cc:260] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.768194 1423 ts_tablet_manager.cc:1428] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250901 01:53:18.773769 685 catalog_manager.cc:5582] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3c774a4edf44d6098a2332e734967a5 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "e3c774a4edf44d6098a2332e734967a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } health_report { overall_health: UNKNOWN } } }
I20250901 01:53:18.774744 1004 tablet_service.cc:1468] Processing CreateTablet for tablet 5274573577374b47b99fa310a5ca83bb (DEFAULT_TABLE table=default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.775249 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5274573577374b47b99fa310a5ca83bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.779578 1146 tablet_service.cc:1468] Processing CreateTablet for tablet 5274573577374b47b99fa310a5ca83bb (DEFAULT_TABLE table=default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.780079 1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5274573577374b47b99fa310a5ca83bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.780987 1439 ts_tablet_manager.cc:1918] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:53:18.781162 1439 log.cc:1199] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal/wals/6263cf06150b49fbbda6c89bd1fb979e
I20250901 01:53:18.781562 1439 ts_tablet_manager.cc:1939] T 6263cf06150b49fbbda6c89bd1fb979e P e3c774a4edf44d6098a2332e734967a5: Deleting consensus metadata
I20250901 01:53:18.785326 673 catalog_manager.cc:4928] TS e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245): tablet 6263cf06150b49fbbda6c89bd1fb979e (table default.table_6bcc2bc2c6574eca8387f75fc422beb9 [id=8ee4e0d7bca3410b9967df0dacf8614d]) successfully deleted
I20250901 01:53:18.787374 1424 tablet_bootstrap.cc:492] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:18.787588 1424 ts_tablet_manager.cc:1397] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.123s user 0.035s sys 0.026s
I20250901 01:53:18.788743 1424 raft_consensus.cc:357] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.789016 1424 raft_consensus.cc:383] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.789129 1424 raft_consensus.cc:738] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.789387 1423 tablet_bootstrap.cc:492] T 5274573577374b47b99fa310a5ca83bb P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:18.789428 1424 consensus_queue.cc:260] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.790002 1424 ts_tablet_manager.cc:1428] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:53:18.790338 1424 tablet_bootstrap.cc:492] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:18.792680 1286 tablet_service.cc:1468] Processing CreateTablet for tablet 5274573577374b47b99fa310a5ca83bb (DEFAULT_TABLE table=default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.793140 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5274573577374b47b99fa310a5ca83bb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:53:18.794589 1496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:18.795194 1423 tablet_bootstrap.cc:654] T 5274573577374b47b99fa310a5ca83bb P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.797729 1423 tablet_bootstrap.cc:492] T 5274573577374b47b99fa310a5ca83bb P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:18.798007 1423 ts_tablet_manager.cc:1397] T 5274573577374b47b99fa310a5ca83bb P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20250901 01:53:18.798970 1423 raft_consensus.cc:357] T 5274573577374b47b99fa310a5ca83bb P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.799300 1423 raft_consensus.cc:383] T 5274573577374b47b99fa310a5ca83bb P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.799868 1423 raft_consensus.cc:738] T 5274573577374b47b99fa310a5ca83bb P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.800313 1422 tablet_bootstrap.cc:492] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:18.800246 1423 consensus_queue.cc:260] T 5274573577374b47b99fa310a5ca83bb P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.800493 1422 ts_tablet_manager.cc:1397] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.130s user 0.004s sys 0.004s
I20250901 01:53:18.792783 1424 tablet_bootstrap.cc:654] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.801949 1422 raft_consensus.cc:357] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.802232 1422 raft_consensus.cc:383] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.802918 1422 raft_consensus.cc:738] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.803253 1422 consensus_queue.cc:260] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.805806 1422 ts_tablet_manager.cc:1428] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250901 01:53:18.806190 1422 tablet_bootstrap.cc:492] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:18.806354 1424 tablet_bootstrap.cc:492] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:18.806569 1424 ts_tablet_manager.cc:1397] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.001s
I20250901 01:53:18.807659 1424 raft_consensus.cc:357] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.807909 1424 raft_consensus.cc:383] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.807993 1424 raft_consensus.cc:738] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.808279 1424 consensus_queue.cc:260] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
W20250901 01:53:18.808979 1495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:18.810143 1424 ts_tablet_manager.cc:1428] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:53:18.811791 1422 tablet_bootstrap.cc:654] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.816159 1424 tablet_bootstrap.cc:492] T 5274573577374b47b99fa310a5ca83bb P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:18.817890 1428 server_base.cc:1047] running on GCE node
I20250901 01:53:18.818001 1424 tablet_bootstrap.cc:654] T 5274573577374b47b99fa310a5ca83bb P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.818178 1423 ts_tablet_manager.cc:1428] T 5274573577374b47b99fa310a5ca83bb P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250901 01:53:18.829031 685 catalog_manager.cc:5582] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e reported cstate change: term changed from 0 to 1, leader changed from <none> to ad2ba2450480454388723cd729cc438e (127.0.104.3). New cstate: current_term: 1 leader_uuid: "ad2ba2450480454388723cd729cc438e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } health_report { overall_health: HEALTHY } } }
I20250901 01:53:18.840251 685 catalog_manager.cc:5582] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 reported cstate change: term changed from 0 to 1, leader changed from <none> to b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b65b3154aefd49f7a37f70e9207e9853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } health_report { overall_health: HEALTHY } } }
I20250901 01:53:18.841065 685 catalog_manager.cc:5582] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 reported cstate change: term changed from 0 to 1, leader changed from <none> to b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b65b3154aefd49f7a37f70e9207e9853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } health_report { overall_health: UNKNOWN } } }
W20250901 01:53:18.865872 1501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:18.867239 1428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:53:18.871155 1428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:53:18.879891 1428 hybrid_clock.cc:648] HybridClock initialized: now 1756691598879683 us; error 193 us; skew 500 ppm
I20250901 01:53:18.880520 1428 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:53:18.880728 685 master_service.cc:948] ReplaceTablet: received request to replace tablet c6380a3fefba4c4caed049b434425733 from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:18.893779 1479 raft_consensus.cc:491] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:18.894047 1479 raft_consensus.cc:513] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.895133 1479 leader_election.cc:290] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2ba2450480454388723cd729cc438e (127.0.104.3:42927), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:18.895756 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a774cb6ccef24d958ba6c898c5e4786c" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:18.896029 1307 raft_consensus.cc:2466] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:18.896358 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a774cb6ccef24d958ba6c898c5e4786c" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:18.896574 1167 raft_consensus.cc:2466] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:18.896569 954 leader_election.cc:304] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters:
I20250901 01:53:18.897497 1479 raft_consensus.cc:2802] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:18.897612 1479 raft_consensus.cc:491] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:18.897768 1479 raft_consensus.cc:3058] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.898469 685 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Sending DeleteTablet for 3 replicas of tablet c6380a3fefba4c4caed049b434425733
I20250901 01:53:18.900388 1146 tablet_service.cc:1515] Processing DeleteTablet for tablet c6380a3fefba4c4caed049b434425733 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f913036f34345fb8b20b3c9f431979e) from {username='slave'} at 127.0.0.1:53914
I20250901 01:53:18.900388 1286 tablet_service.cc:1515] Processing DeleteTablet for tablet c6380a3fefba4c4caed049b434425733 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f913036f34345fb8b20b3c9f431979e) from {username='slave'} at 127.0.0.1:58732
I20250901 01:53:18.900696 1439 tablet_replica.cc:331] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5: stopping tablet replica
I20250901 01:53:18.900724 1004 tablet_service.cc:1515] Processing DeleteTablet for tablet c6380a3fefba4c4caed049b434425733 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f913036f34345fb8b20b3c9f431979e) from {username='slave'} at 127.0.0.1:35622
I20250901 01:53:18.900895 1439 raft_consensus.cc:2241] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:18.900975 1440 tablet_replica.cc:331] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e: stopping tablet replica
I20250901 01:53:18.901113 1439 raft_consensus.cc:2270] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:18.901186 1440 raft_consensus.cc:2241] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:53:18.901435 1440 pending_rounds.cc:70] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e: Trying to abort 1 pending ops.
I20250901 01:53:18.901512 1440 pending_rounds.cc:77] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408787113934848 op_type: NO_OP noop_request { }
I20250901 01:53:18.901616 1440 raft_consensus.cc:2887] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:53:18.901724 1440 raft_consensus.cc:2270] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:18.901728 1439 ts_tablet_manager.cc:1905] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:18.904457 1438 tablet_replica.cc:331] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853: stopping tablet replica
I20250901 01:53:18.904781 1438 raft_consensus.cc:2241] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:18.905035 1438 raft_consensus.cc:2270] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:18.906514 1438 ts_tablet_manager.cc:1905] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:18.906605 1440 ts_tablet_manager.cc:1905] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:18.911530 685 catalog_manager.cc:6652] ReplaceTablet: tablet c6380a3fefba4c4caed049b434425733 deleted and replaced by tablet 8f913036f34345fb8b20b3c9f431979e
I20250901 01:53:18.911957 685 master_service.cc:948] ReplaceTablet: received request to replace tablet c6380a3fefba4c4caed049b434425733 from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:18.912530 685 master_service.cc:948] ReplaceTablet: received request to replace tablet a3c5c349f8f8435f8b125a6d527f970a from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:18.916349 1479 raft_consensus.cc:513] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.917912 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a774cb6ccef24d958ba6c898c5e4786c" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:18.918566 1307 raft_consensus.cc:3058] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.921000 1439 ts_tablet_manager.cc:1918] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:53:18.921175 1439 log.cc:1199] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal/wals/c6380a3fefba4c4caed049b434425733
I20250901 01:53:18.922209 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a774cb6ccef24d958ba6c898c5e4786c" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:18.922474 1167 raft_consensus.cc:3058] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:18.926347 1307 raft_consensus.cc:2466] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:18.926406 1167 raft_consensus.cc:2466] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:18.926898 954 leader_election.cc:304] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters:
I20250901 01:53:18.927295 1478 raft_consensus.cc:2802] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:18.927338 1479 leader_election.cc:290] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Requested vote from peers ad2ba2450480454388723cd729cc438e (127.0.104.3:42927), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:18.927470 1478 raft_consensus.cc:695] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Becoming Leader. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Running, Role: LEADER
I20250901 01:53:18.927763 1478 consensus_queue.cc:237] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } }
I20250901 01:53:18.931136 1146 tablet_service.cc:1468] Processing CreateTablet for tablet 8f913036f34345fb8b20b3c9f431979e (DEFAULT_TABLE table=default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.931298 1440 ts_tablet_manager.cc:1918] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:18.931484 1440 log.cc:1199] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal/wals/c6380a3fefba4c4caed049b434425733
I20250901 01:53:18.931695 1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f913036f34345fb8b20b3c9f431979e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.932009 1440 ts_tablet_manager.cc:1939] T c6380a3fefba4c4caed049b434425733 P ad2ba2450480454388723cd729cc438e: Deleting consensus metadata
I20250901 01:53:18.932126 1004 tablet_service.cc:1468] Processing CreateTablet for tablet 8f913036f34345fb8b20b3c9f431979e (DEFAULT_TABLE table=default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.932559 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f913036f34345fb8b20b3c9f431979e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.933003 1439 ts_tablet_manager.cc:1939] T c6380a3fefba4c4caed049b434425733 P e3c774a4edf44d6098a2332e734967a5: Deleting consensus metadata
I20250901 01:53:18.915230 685 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_795876dee30e438da68abc8a5bbae13f" } modify_external_catalogs: true
I20250901 01:53:18.931496 1286 tablet_service.cc:1468] Processing CreateTablet for tablet 8f913036f34345fb8b20b3c9f431979e (DEFAULT_TABLE table=default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:18.934329 685 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_795876dee30e438da68abc8a5bbae13f" } modify_external_catalogs: true
I20250901 01:53:18.934680 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f913036f34345fb8b20b3c9f431979e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:18.938822 673 catalog_manager.cc:4928] TS e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245): tablet c6380a3fefba4c4caed049b434425733 (table default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]) successfully deleted
I20250901 01:53:18.939765 674 catalog_manager.cc:4928] TS ad2ba2450480454388723cd729cc438e (127.0.104.3:42927): tablet c6380a3fefba4c4caed049b434425733 (table default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]) successfully deleted
I20250901 01:53:18.955099 1423 tablet_bootstrap.cc:492] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:18.957683 1423 tablet_bootstrap.cc:654] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:18.963362 1423 tablet_bootstrap.cc:492] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:18.963578 1423 ts_tablet_manager.cc:1397] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20250901 01:53:18.964417 1423 raft_consensus.cc:357] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.964638 1423 raft_consensus.cc:383] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.964730 1423 raft_consensus.cc:738] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.965001 1423 consensus_queue.cc:260] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.968505 1422 tablet_bootstrap.cc:492] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:18.968713 1422 ts_tablet_manager.cc:1397] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.163s user 0.020s sys 0.031s
I20250901 01:53:18.969208 1423 ts_tablet_manager.cc:1428] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20250901 01:53:18.970744 1424 tablet_bootstrap.cc:492] T 5274573577374b47b99fa310a5ca83bb P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:18.970942 1424 ts_tablet_manager.cc:1397] T 5274573577374b47b99fa310a5ca83bb P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.155s user 0.019s sys 0.035s
I20250901 01:53:18.972673 1422 raft_consensus.cc:357] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.972903 1422 raft_consensus.cc:383] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.972987 1422 raft_consensus.cc:738] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.974071 1438 ts_tablet_manager.cc:1918] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:53:18.974578 1424 raft_consensus.cc:357] T 5274573577374b47b99fa310a5ca83bb P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.974782 1424 raft_consensus.cc:383] T 5274573577374b47b99fa310a5ca83bb P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:18.974892 1424 raft_consensus.cc:738] T 5274573577374b47b99fa310a5ca83bb P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:18.975836 1438 log.cc:1199] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/wals/c6380a3fefba4c4caed049b434425733
I20250901 01:53:18.976666 1422 consensus_queue.cc:260] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.977142 1424 consensus_queue.cc:260] T 5274573577374b47b99fa310a5ca83bb P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:18.977893 1424 ts_tablet_manager.cc:1428] T 5274573577374b47b99fa310a5ca83bb P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250901 01:53:18.978320 1424 tablet_bootstrap.cc:492] T 8f913036f34345fb8b20b3c9f431979e P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:18.985733 1438 ts_tablet_manager.cc:1939] T c6380a3fefba4c4caed049b434425733 P b65b3154aefd49f7a37f70e9207e9853: Deleting consensus metadata
I20250901 01:53:18.985965 687 catalog_manager.cc:3460] renamed table default.table_fee1d3b07998429a91772db3deb1ea39 [id=b449add8498f4d1fbb05b28cc933c647] in HMS: new name default.table_d1c05496169b472fa9b021013c4afd01
I20250901 01:53:18.987447 676 catalog_manager.cc:4928] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): tablet c6380a3fefba4c4caed049b434425733 (table default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]) successfully deleted
I20250901 01:53:18.989894 1422 ts_tablet_manager.cc:1428] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20250901 01:53:18.990311 1422 tablet_bootstrap.cc:492] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:18.993445 1424 tablet_bootstrap.cc:654] T 8f913036f34345fb8b20b3c9f431979e P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
W20250901 01:53:18.995404 689 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_61dfd07a8f784854b5d9e104e010daaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:53:18.996057 1422 tablet_bootstrap.cc:654] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
W20250901 01:53:18.996680 682 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c95fbf8fa7a64aa18002f1e268cdea68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:18.997583 680 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5178a246373c4a26a1a2e11a77c7e56c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:53:18.997622 1428 webserver.cc:480] Webserver started at http://127.0.104.60:35767/ using document root <none> and password file <none>
W20250901 01:53:18.998684 684 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_da487bf0d57f439bb9da2ec57880a73b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:53:18.999197 1428 fs_manager.cc:362] Metadata directory not provided
I20250901 01:53:18.999423 1428 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 01:53:19.004264 1422 tablet_bootstrap.cc:492] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:19.004513 1422 ts_tablet_manager.cc:1397] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20250901 01:53:19.005764 1422 raft_consensus.cc:357] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.006039 1422 raft_consensus.cc:383] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:19.006140 1422 raft_consensus.cc:738] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:19.006498 1422 consensus_queue.cc:260] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.010296 1422 ts_tablet_manager.cc:1428] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20250901 01:53:19.015594 1422 tablet_bootstrap.cc:492] T 8f913036f34345fb8b20b3c9f431979e P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:19.022672 687 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:58182 (request call id 28) took 1377 ms. Trace:
I20250901 01:53:19.023026 1422 tablet_bootstrap.cc:654] T 8f913036f34345fb8b20b3c9f431979e P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:19.022836 687 rpcz_store.cc:276] 0901 01:53:17.644906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:17.645313 (+ 407us) service_pool.cc:225] Handling call
0901 01:53:18.681681 (+1036368us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:53:19.021814 (+340133us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:53:19.022257 (+ 443us) catalog_manager.cc:3683] Apply alter schema
0901 01:53:19.022622 (+ 365us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":3829,"Hive Metastore.run_cpu_time_us":1282,"Hive Metastore.run_wall_time_us":299853}
I20250901 01:53:19.025686 687 catalog_manager.cc:5582] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 reported cstate change: term changed from 0 to 1, leader changed from <none> to b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b65b3154aefd49f7a37f70e9207e9853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } health_report { overall_health: UNKNOWN } } }
I20250901 01:53:19.030063 1424 tablet_bootstrap.cc:492] T 8f913036f34345fb8b20b3c9f431979e P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:19.030335 1424 ts_tablet_manager.cc:1397] T 8f913036f34345fb8b20b3c9f431979e P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.019s
I20250901 01:53:19.031129 1424 raft_consensus.cc:357] T 8f913036f34345fb8b20b3c9f431979e P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.031385 1424 raft_consensus.cc:383] T 8f913036f34345fb8b20b3c9f431979e P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:19.031463 1424 raft_consensus.cc:738] T 8f913036f34345fb8b20b3c9f431979e P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:19.033955 1428 fs_manager.cc:714] Time spent opening directory manager: real 0.016s user 0.007s sys 0.000s
I20250901 01:53:19.033891 1424 consensus_queue.cc:260] T 8f913036f34345fb8b20b3c9f431979e P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.035063 1424 ts_tablet_manager.cc:1428] T 8f913036f34345fb8b20b3c9f431979e P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250901 01:53:19.035202 1422 tablet_bootstrap.cc:492] T 8f913036f34345fb8b20b3c9f431979e P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:19.035418 1422 ts_tablet_manager.cc:1397] T 8f913036f34345fb8b20b3c9f431979e P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20250901 01:53:19.036511 1422 raft_consensus.cc:357] T 8f913036f34345fb8b20b3c9f431979e P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.036810 1422 raft_consensus.cc:383] T 8f913036f34345fb8b20b3c9f431979e P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:19.036885 1422 raft_consensus.cc:738] T 8f913036f34345fb8b20b3c9f431979e P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:19.037166 1422 consensus_queue.cc:260] T 8f913036f34345fb8b20b3c9f431979e P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.041754 1422 ts_tablet_manager.cc:1428] T 8f913036f34345fb8b20b3c9f431979e P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250901 01:53:19.055229 1307 raft_consensus.cc:1273] T 586fbc1ef054435aaaea9ae961f15115 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Refusing update from remote peer e3c774a4edf44d6098a2332e734967a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 01:53:19.056708 1429 consensus_queue.cc:1035] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:19.063592 687 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_9f3451ed25714464ad16b3e94214d0e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:19.066380 1515 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:19.066409 1024 raft_consensus.cc:1273] T 586fbc1ef054435aaaea9ae961f15115 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Refusing update from remote peer e3c774a4edf44d6098a2332e734967a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 01:53:19.067541 1429 consensus_queue.cc:1035] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:19.068753 1428 fs_manager.cc:730] Time spent opening block manager: real 0.018s user 0.010s sys 0.005s
I20250901 01:53:19.068985 1428 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/data,/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/wal
uuid: "835a442fbc4c48eb9148ca677e410a59"
format_stamp: "Formatted at 2025-09-01 01:53:14 on dist-test-slave-r4vn"
I20250901 01:53:19.070060 1428 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:53:19.092674 1512 mvcc.cc:204] Tried to move back new op lower bound from 7195408789716008960 to 7195408788060442624. Current Snapshot: MvccSnapshot[applied={T|T < 7195408789716008960}]
I20250901 01:53:19.093256 1516 mvcc.cc:204] Tried to move back new op lower bound from 7195408789716008960 to 7195408788060442624. Current Snapshot: MvccSnapshot[applied={T|T < 7195408789716008960}]
I20250901 01:53:19.103655 1518 mvcc.cc:204] Tried to move back new op lower bound from 7195408789716008960 to 7195408788060442624. Current Snapshot: MvccSnapshot[applied={T|T < 7195408789716008960}]
I20250901 01:53:19.131472 1522 tablet.cc:1722] T 586fbc1ef054435aaaea9ae961f15115 P ad2ba2450480454388723cd729cc438e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:53:19.132664 1521 tablet.cc:1722] T 586fbc1ef054435aaaea9ae961f15115 P e3c774a4edf44d6098a2332e734967a5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:53:19.137068 1523 tablet.cc:1722] T 586fbc1ef054435aaaea9ae961f15115 P b65b3154aefd49f7a37f70e9207e9853: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:53:19.162079 1428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 01:53:19.163379 1428 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:53:19.163695 1428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:53:19.195161 1450 raft_consensus.cc:491] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:19.195410 1450 raft_consensus.cc:513] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.196218 1450 leader_election.cc:290] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:19.197495 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5274573577374b47b99fa310a5ca83bb" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:19.197774 1307 raft_consensus.cc:2466] T 5274573577374b47b99fa310a5ca83bb P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 0.
I20250901 01:53:19.198751 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5274573577374b47b99fa310a5ca83bb" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:19.199000 1024 raft_consensus.cc:2466] T 5274573577374b47b99fa310a5ca83bb P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 0.
I20250901 01:53:19.199018 1101 leader_election.cc:304] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:19.199875 1450 raft_consensus.cc:2802] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:19.200001 1450 raft_consensus.cc:491] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:19.200085 1450 raft_consensus.cc:3058] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.203392 1450 raft_consensus.cc:513] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.204713 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5274573577374b47b99fa310a5ca83bb" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:19.204890 681 catalog_manager.cc:3460] renamed table default.table_e628b8e1d2214e458c8a283659d45a07 [id=f1db819ca90642eaac30e77fd5f5c8d5] in HMS: new name default.table_7e006da3e6d24d309af545afeac11ca4
I20250901 01:53:19.204962 1024 raft_consensus.cc:3058] T 5274573577374b47b99fa310a5ca83bb P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.205406 1450 leader_election.cc:290] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Requested vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:19.206283 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5274573577374b47b99fa310a5ca83bb" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:19.206512 1307 raft_consensus.cc:3058] T 5274573577374b47b99fa310a5ca83bb P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.207930 1024 raft_consensus.cc:2466] T 5274573577374b47b99fa310a5ca83bb P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 1.
I20250901 01:53:19.208494 1103 leader_election.cc:304] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:19.208876 1450 raft_consensus.cc:2802] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:19.210099 1307 raft_consensus.cc:2466] T 5274573577374b47b99fa310a5ca83bb P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 1.
I20250901 01:53:19.210799 1450 raft_consensus.cc:695] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [term 1 LEADER]: Becoming Leader. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Running, Role: LEADER
I20250901 01:53:19.211123 1450 consensus_queue.cc:237] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.308374 1467 consensus_queue.cc:1035] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:19.312883 688 catalog_manager.cc:3460] renamed table default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8] in HMS: new name default.table_8f21a26e011c4ed1b1ae7cb364e09195
I20250901 01:53:19.337544 1561 consensus_queue.cc:1035] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250901 01:53:19.359517 1450 raft_consensus.cc:491] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:19.359787 1450 raft_consensus.cc:513] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.360715 1450 leader_election.cc:290] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:19.361724 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "138d19e14b734c1cace0d7ba51ab2caa" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:19.362002 1307 raft_consensus.cc:2466] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 0.
I20250901 01:53:19.363030 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "138d19e14b734c1cace0d7ba51ab2caa" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:19.363904 1024 raft_consensus.cc:2466] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 0.
I20250901 01:53:19.364686 1101 leader_election.cc:304] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:19.365370 1450 raft_consensus.cc:2802] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:19.365484 1450 raft_consensus.cc:491] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:19.365602 1450 raft_consensus.cc:3058] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.369158 1450 raft_consensus.cc:513] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.370813 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "138d19e14b734c1cace0d7ba51ab2caa" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:19.371057 1024 raft_consensus.cc:3058] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.374728 1024 raft_consensus.cc:2466] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 1.
I20250901 01:53:19.375378 1103 leader_election.cc:304] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:19.375819 1429 raft_consensus.cc:2802] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:19.375977 1429 raft_consensus.cc:695] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [term 1 LEADER]: Becoming Leader. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Running, Role: LEADER
I20250901 01:53:19.376330 1429 consensus_queue.cc:237] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.391513 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "138d19e14b734c1cace0d7ba51ab2caa" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:19.391820 1307 raft_consensus.cc:3058] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.395440 1307 raft_consensus.cc:2466] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 1.
I20250901 01:53:19.396354 1450 leader_election.cc:290] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Requested vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:19.407764 689 catalog_manager.cc:2149] created HMS catalog entry for table [id=ae0da2fb2d1542308ccb9a95247c1d98]
I20250901 01:53:19.414800 689 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58182 (request call id 35) took 1569 ms. Trace:
I20250901 01:53:19.414958 689 rpcz_store.cc:276] 0901 01:53:17.845239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:18.043319 (+198080us) service_pool.cc:225] Handling call
0901 01:53:18.995562 (+952243us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:53:18.996127 (+ 565us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:53:19.407704 (+411577us) catalog_manager.cc:2148] Created new table in HMS catalog
0901 01:53:19.414524 (+ 6820us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:53:19.414640 (+ 116us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:53:19.414752 (+ 112us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:53:19.408622 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:53:19.408754 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:53:19.408761 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:53:19.408764 (+ 3us) tablet.cc:660] Decoding operations
0901 01:53:19.408864 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:53:19.408883 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0901 01:53:19.408888 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:53:19.408978 (+ 90us) tablet.cc:699] Row locks acquired
0901 01:53:19.408981 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:53:19.409032 (+ 51us) write_op.cc:270] Start()
0901 01:53:19.409071 (+ 39us) write_op.cc:276] Timestamp: P: 1756691599409015 usec, L: 0
0901 01:53:19.409075 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:53:19.409313 (+ 238us) log.cc:844] Serialized 432 byte log entry
0901 01:53:19.413037 (+ 3724us) op_driver.cc:464] REPLICATION: finished
0901 01:53:19.413197 (+ 160us) write_op.cc:301] APPLY: starting
0901 01:53:19.413215 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0901 01:53:19.413269 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0901 01:53:19.413295 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:53:19.413407 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:53:19.413443 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:53:19.413451 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:53:19.413533 (+ 82us) log.cc:844] Serialized 28 byte log entry
0901 01:53:19.413597 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:19.413621 (+ 24us) write_op.cc:454] Released schema lock
0901 01:53:19.413627 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:53:19.413698 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:19.413702 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":316274,"Hive Metastore.run_cpu_time_us":335,"Hive Metastore.run_wall_time_us":94775,"child_traces":[["op",{"apply.queue_time_us":104,"apply.run_cpu_time_us":495,"apply.run_wall_time_us":495,"num_ops":2,"prepare.queue_time_us":109,"prepare.run_cpu_time_us":819,"prepare.run_wall_time_us":820,"raft.queue_time_us":1017,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":367,"replication_time_us":3933}]]}
I20250901 01:53:19.416922 689 catalog_manager.cc:5674] TS ad2ba2450480454388723cd729cc438e (127.0.104.3:42927) does not have the latest schema for tablet 586fbc1ef054435aaaea9ae961f15115 (table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647]). Expected version 1 got 0
I20250901 01:53:19.430518 1286 tablet_service.cc:1468] Processing CreateTablet for tablet 8dd9f23bffa543a490c018108ad41051 (DEFAULT_TABLE table=default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:19.430735 1004 tablet_service.cc:1468] Processing CreateTablet for tablet 8dd9f23bffa543a490c018108ad41051 (DEFAULT_TABLE table=default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:19.430964 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dd9f23bffa543a490c018108ad41051. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:19.431229 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dd9f23bffa543a490c018108ad41051. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:19.433809 1146 tablet_service.cc:1468] Processing CreateTablet for tablet 8dd9f23bffa543a490c018108ad41051 (DEFAULT_TABLE table=default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:19.434273 1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dd9f23bffa543a490c018108ad41051. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:19.437196 1424 tablet_bootstrap.cc:492] T 8dd9f23bffa543a490c018108ad41051 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:19.439805 1424 tablet_bootstrap.cc:654] T 8dd9f23bffa543a490c018108ad41051 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:19.443259 689 catalog_manager.cc:5674] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315) does not have the latest schema for tablet 586fbc1ef054435aaaea9ae961f15115 (table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647]). Expected version 1 got 0
I20250901 01:53:19.445636 689 catalog_manager.cc:5674] TS e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245) does not have the latest schema for tablet 586fbc1ef054435aaaea9ae961f15115 (table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647]). Expected version 1 got 0
I20250901 01:53:19.446101 1423 tablet_bootstrap.cc:492] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:19.448110 1423 tablet_bootstrap.cc:654] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:19.454314 1422 tablet_bootstrap.cc:492] T 8dd9f23bffa543a490c018108ad41051 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:19.456868 1422 tablet_bootstrap.cc:654] T 8dd9f23bffa543a490c018108ad41051 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:19.456969 1478 consensus_queue.cc:1035] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:19.471210 1422 tablet_bootstrap.cc:492] T 8dd9f23bffa543a490c018108ad41051 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:19.471428 1422 ts_tablet_manager.cc:1397] T 8dd9f23bffa543a490c018108ad41051 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.009s
I20250901 01:53:19.472397 1422 raft_consensus.cc:357] T 8dd9f23bffa543a490c018108ad41051 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.472714 1422 raft_consensus.cc:383] T 8dd9f23bffa543a490c018108ad41051 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:19.472815 1422 raft_consensus.cc:738] T 8dd9f23bffa543a490c018108ad41051 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:19.473110 1422 consensus_queue.cc:260] T 8dd9f23bffa543a490c018108ad41051 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.474902 1423 tablet_bootstrap.cc:492] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:19.475162 1423 ts_tablet_manager.cc:1397] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.004s
I20250901 01:53:19.476038 1423 raft_consensus.cc:357] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.476389 1423 raft_consensus.cc:383] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:19.476540 1423 raft_consensus.cc:738] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:19.476857 1423 consensus_queue.cc:260] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.479791 1422 ts_tablet_manager.cc:1428] T 8dd9f23bffa543a490c018108ad41051 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20250901 01:53:19.481202 1423 ts_tablet_manager.cc:1428] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250901 01:53:19.481700 689 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647] alter complete (version 1)
I20250901 01:53:19.483035 1424 tablet_bootstrap.cc:492] T 8dd9f23bffa543a490c018108ad41051 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:19.483210 1424 ts_tablet_manager.cc:1397] T 8dd9f23bffa543a490c018108ad41051 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.007s
I20250901 01:53:19.484438 1424 raft_consensus.cc:357] T 8dd9f23bffa543a490c018108ad41051 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.484637 1424 raft_consensus.cc:383] T 8dd9f23bffa543a490c018108ad41051 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:19.484730 1424 raft_consensus.cc:738] T 8dd9f23bffa543a490c018108ad41051 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:19.484984 1424 consensus_queue.cc:260] T 8dd9f23bffa543a490c018108ad41051 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.485584 1424 ts_tablet_manager.cc:1428] T 8dd9f23bffa543a490c018108ad41051 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:53:19.488390 1478 raft_consensus.cc:491] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:19.488551 1478 raft_consensus.cc:513] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.489205 1478 leader_election.cc:290] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:19.490492 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dd9f23bffa543a490c018108ad41051" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:19.490741 1307 raft_consensus.cc:2466] T 8dd9f23bffa543a490c018108ad41051 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:19.490881 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dd9f23bffa543a490c018108ad41051" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:19.491099 1167 raft_consensus.cc:2466] T 8dd9f23bffa543a490c018108ad41051 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:19.491466 954 leader_election.cc:304] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters:
I20250901 01:53:19.494885 1478 raft_consensus.cc:2802] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:19.495025 1478 raft_consensus.cc:491] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:19.495115 1478 raft_consensus.cc:3058] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.498258 1478 raft_consensus.cc:513] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.498960 1478 leader_election.cc:290] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Requested vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:19.499300 1166 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dd9f23bffa543a490c018108ad41051" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:19.499507 1166 raft_consensus.cc:3058] T 8dd9f23bffa543a490c018108ad41051 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.501673 689 catalog_manager.cc:5582] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3c774a4edf44d6098a2332e734967a5 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "e3c774a4edf44d6098a2332e734967a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } health_report { overall_health: UNKNOWN } } }
I20250901 01:53:19.502238 689 catalog_manager.cc:5582] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3c774a4edf44d6098a2332e734967a5 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "e3c774a4edf44d6098a2332e734967a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } health_report { overall_health: UNKNOWN } } }
I20250901 01:53:19.503216 1166 raft_consensus.cc:2466] T 8dd9f23bffa543a490c018108ad41051 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:19.503777 953 leader_election.cc:304] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:19.506666 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dd9f23bffa543a490c018108ad41051" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:19.506915 1307 raft_consensus.cc:3058] T 8dd9f23bffa543a490c018108ad41051 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.510660 1478 raft_consensus.cc:2802] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:19.510756 1307 raft_consensus.cc:2466] T 8dd9f23bffa543a490c018108ad41051 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:19.511931 1577 consensus_queue.cc:1035] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:19.514129 1478 raft_consensus.cc:695] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Becoming Leader. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Running, Role: LEADER
I20250901 01:53:19.514513 1478 consensus_queue.cc:237] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.520401 1577 raft_consensus.cc:491] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:19.520758 1577 raft_consensus.cc:513] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.522392 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f913036f34345fb8b20b3c9f431979e" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:19.522720 1167 raft_consensus.cc:2466] T 8f913036f34345fb8b20b3c9f431979e P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:19.523435 953 leader_election.cc:304] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:19.524278 1478 raft_consensus.cc:2802] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:19.524418 1478 raft_consensus.cc:491] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:19.524528 1478 raft_consensus.cc:3058] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.524668 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f913036f34345fb8b20b3c9f431979e" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:19.524953 1307 raft_consensus.cc:2466] T 8f913036f34345fb8b20b3c9f431979e P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:19.525497 1577 leader_election.cc:290] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:19.526762 682 catalog_manager.cc:2149] created HMS catalog entry for table [id=d830537abb3c488ca7b925a8a491432a]
I20250901 01:53:19.527433 1478 raft_consensus.cc:513] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.528028 1478 leader_election.cc:290] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Requested vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:19.529078 1306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f913036f34345fb8b20b3c9f431979e" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:19.529320 1306 raft_consensus.cc:3058] T 8f913036f34345fb8b20b3c9f431979e P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.533201 1306 raft_consensus.cc:2466] T 8f913036f34345fb8b20b3c9f431979e P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:19.534965 954 leader_election.cc:304] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters:
I20250901 01:53:19.535620 1577 consensus_queue.cc:1035] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:19.538183 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f913036f34345fb8b20b3c9f431979e" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:19.538308 1478 raft_consensus.cc:2802] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:19.538435 1167 raft_consensus.cc:3058] T 8f913036f34345fb8b20b3c9f431979e P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.538479 1478 raft_consensus.cc:695] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Becoming Leader. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Running, Role: LEADER
I20250901 01:53:19.538779 1478 consensus_queue.cc:237] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.544941 682 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58182 (request call id 36) took 1629 ms. Trace:
I20250901 01:53:19.545632 1167 raft_consensus.cc:2466] T 8f913036f34345fb8b20b3c9f431979e P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:19.547365 682 rpcz_store.cc:276] 0901 01:53:17.915740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:18.046793 (+131053us) service_pool.cc:225] Handling call
0901 01:53:18.996774 (+949981us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:53:18.997142 (+ 368us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:53:19.526709 (+529567us) catalog_manager.cc:2148] Created new table in HMS catalog
0901 01:53:19.544048 (+ 17339us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:53:19.544755 (+ 707us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:53:19.544894 (+ 139us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:53:19.531845 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:53:19.532008 (+ 163us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:53:19.532018 (+ 10us) write_op.cc:435] Acquired schema lock
0901 01:53:19.532023 (+ 5us) tablet.cc:660] Decoding operations
0901 01:53:19.532164 (+ 141us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:53:19.532179 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0901 01:53:19.532186 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:53:19.532295 (+ 109us) tablet.cc:699] Row locks acquired
0901 01:53:19.532299 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:53:19.532340 (+ 41us) write_op.cc:270] Start()
0901 01:53:19.532382 (+ 42us) write_op.cc:276] Timestamp: P: 1756691599532325 usec, L: 0
0901 01:53:19.532388 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:53:19.532620 (+ 232us) log.cc:844] Serialized 432 byte log entry
0901 01:53:19.542822 (+ 10202us) op_driver.cc:464] REPLICATION: finished
0901 01:53:19.543014 (+ 192us) write_op.cc:301] APPLY: starting
0901 01:53:19.543039 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0901 01:53:19.543117 (+ 78us) tablet.cc:1368] finished BulkCheckPresence
0901 01:53:19.543192 (+ 75us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:53:19.543342 (+ 150us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:53:19.543391 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:53:19.543403 (+ 12us) write_op.cc:312] APPLY: finished
0901 01:53:19.543505 (+ 102us) log.cc:844] Serialized 28 byte log entry
0901 01:53:19.543585 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:19.543615 (+ 30us) write_op.cc:454] Released schema lock
0901 01:53:19.543625 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0901 01:53:19.543718 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:19.543722 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":410232,"Hive Metastore.run_cpu_time_us":279,"Hive Metastore.run_wall_time_us":118981,"child_traces":[["op",{"apply.queue_time_us":125,"apply.run_cpu_time_us":692,"apply.run_wall_time_us":691,"mutex_wait_us":4308,"num_ops":2,"prepare.queue_time_us":4393,"prepare.run_cpu_time_us":896,"prepare.run_wall_time_us":897,"raft.queue_time_us":72,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":10385}]]}
I20250901 01:53:19.554731 682 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_c2864548fe1a4719b73cf55d0933e538" } modify_external_catalogs: true
I20250901 01:53:19.555109 682 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_c2864548fe1a4719b73cf55d0933e538" } modify_external_catalogs: true
I20250901 01:53:19.567979 1004 tablet_service.cc:1468] Processing CreateTablet for tablet 4e1a5e7ec3f842ffa6f24e5b315eb6dd (DEFAULT_TABLE table=default.table_c95fbf8fa7a64aa18002f1e268cdea68 [id=d830537abb3c488ca7b925a8a491432a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:19.568296 1146 tablet_service.cc:1468] Processing CreateTablet for tablet 4e1a5e7ec3f842ffa6f24e5b315eb6dd (DEFAULT_TABLE table=default.table_c95fbf8fa7a64aa18002f1e268cdea68 [id=d830537abb3c488ca7b925a8a491432a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:19.568567 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e1a5e7ec3f842ffa6f24e5b315eb6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:19.568735 1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e1a5e7ec3f842ffa6f24e5b315eb6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:19.572371 1286 tablet_service.cc:1468] Processing CreateTablet for tablet 4e1a5e7ec3f842ffa6f24e5b315eb6dd (DEFAULT_TABLE table=default.table_c95fbf8fa7a64aa18002f1e268cdea68 [id=d830537abb3c488ca7b925a8a491432a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:19.572826 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e1a5e7ec3f842ffa6f24e5b315eb6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:19.575186 1422 tablet_bootstrap.cc:492] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:19.577580 1422 tablet_bootstrap.cc:654] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:19.584941 689 catalog_manager.cc:5582] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 reported cstate change: term changed from 0 to 1, leader changed from <none> to b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b65b3154aefd49f7a37f70e9207e9853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } health_report { overall_health: UNKNOWN } } }
I20250901 01:53:19.587199 1578 consensus_queue.cc:1035] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:19.592494 1424 tablet_bootstrap.cc:492] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:19.593111 689 catalog_manager.cc:5582] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 reported cstate change: term changed from 0 to 1, leader changed from <none> to b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b65b3154aefd49f7a37f70e9207e9853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } health_report { overall_health: UNKNOWN } } }
I20250901 01:53:19.594798 1424 tablet_bootstrap.cc:654] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:19.599550 1423 tablet_bootstrap.cc:492] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:19.601572 1423 tablet_bootstrap.cc:654] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:19.602792 1424 tablet_bootstrap.cc:492] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:19.602989 1424 ts_tablet_manager.cc:1397] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20250901 01:53:19.604223 1424 raft_consensus.cc:357] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.604526 1424 raft_consensus.cc:383] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:19.604612 1424 raft_consensus.cc:738] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:19.604924 1424 consensus_queue.cc:260] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.607085 1424 ts_tablet_manager.cc:1428] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250901 01:53:19.620419 1423 tablet_bootstrap.cc:492] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:19.620653 1423 ts_tablet_manager.cc:1397] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.001s
I20250901 01:53:19.623751 1423 raft_consensus.cc:357] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.624042 1423 raft_consensus.cc:383] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:19.624140 1423 raft_consensus.cc:738] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:19.624465 1423 consensus_queue.cc:260] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.628787 1423 ts_tablet_manager.cc:1428] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20250901 01:53:19.633128 1422 tablet_bootstrap.cc:492] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:19.633378 1422 ts_tablet_manager.cc:1397] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.008s
I20250901 01:53:19.634338 1422 raft_consensus.cc:357] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.634603 1422 raft_consensus.cc:383] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:19.634727 1422 raft_consensus.cc:738] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:19.635003 1422 consensus_queue.cc:260] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.636206 1422 ts_tablet_manager.cc:1428] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:53:19.645493 689 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_98a9d1400149426bab73e31aea263f5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:19.666280 1428 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:36985
I20250901 01:53:19.666771 1596 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:36985 every 8 connection(s)
I20250901 01:53:19.670377 1428 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-2/data/info.pb
I20250901 01:53:19.678707 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1fHXBS/build/asan/bin/kudu as pid 1428
I20250901 01:53:19.688412 680 catalog_manager.cc:2149] created HMS catalog entry for table [id=bffe8fa63e6045b98694851a029bc0d9]
I20250901 01:53:19.695585 680 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58182 (request call id 37) took 1723 ms. Trace:
I20250901 01:53:19.695763 680 rpcz_store.cc:276] 0901 01:53:17.972206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:18.139185 (+166979us) service_pool.cc:225] Handling call
0901 01:53:18.997666 (+858481us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:53:18.997994 (+ 328us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:53:19.688356 (+690362us) catalog_manager.cc:2148] Created new table in HMS catalog
0901 01:53:19.695164 (+ 6808us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:53:19.695331 (+ 167us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:53:19.695481 (+ 150us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:53:19.689438 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:53:19.689598 (+ 160us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:53:19.689606 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:53:19.689609 (+ 3us) tablet.cc:660] Decoding operations
0901 01:53:19.689724 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:53:19.689739 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0901 01:53:19.689745 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:53:19.689833 (+ 88us) tablet.cc:699] Row locks acquired
0901 01:53:19.689836 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:53:19.689881 (+ 45us) write_op.cc:270] Start()
0901 01:53:19.689921 (+ 40us) write_op.cc:276] Timestamp: P: 1756691599689865 usec, L: 0
0901 01:53:19.689925 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:53:19.690177 (+ 252us) log.cc:844] Serialized 432 byte log entry
0901 01:53:19.694414 (+ 4237us) op_driver.cc:464] REPLICATION: finished
0901 01:53:19.694556 (+ 142us) write_op.cc:301] APPLY: starting
0901 01:53:19.694576 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0901 01:53:19.694629 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0901 01:53:19.694668 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:53:19.694783 (+ 115us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:53:19.694822 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:53:19.694830 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:53:19.694924 (+ 94us) log.cc:844] Serialized 28 byte log entry
0901 01:53:19.694995 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:19.695020 (+ 25us) write_op.cc:454] Released schema lock
0901 01:53:19.695028 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0901 01:53:19.695117 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:19.695121 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":528419,"Hive Metastore.run_cpu_time_us":339,"Hive Metastore.run_wall_time_us":161739,"child_traces":[["op",{"apply.queue_time_us":90,"apply.run_cpu_time_us":550,"apply.run_wall_time_us":550,"num_ops":2,"prepare.queue_time_us":83,"prepare.run_cpu_time_us":892,"prepare.run_wall_time_us":892,"raft.queue_time_us":1242,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":288,"replication_time_us":4451}]]}
I20250901 01:53:19.697042 680 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_832e2227a6314ac2a050f76013e116fc" } modify_external_catalogs: true
I20250901 01:53:19.697373 680 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_832e2227a6314ac2a050f76013e116fc" } modify_external_catalogs: true
I20250901 01:53:19.705922 1004 tablet_service.cc:1468] Processing CreateTablet for tablet ca68c9a0c8fb4cccb081b404daff8b2a (DEFAULT_TABLE table=default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:19.706516 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca68c9a0c8fb4cccb081b404daff8b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:19.706610 1146 tablet_service.cc:1468] Processing CreateTablet for tablet ca68c9a0c8fb4cccb081b404daff8b2a (DEFAULT_TABLE table=default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:19.707038 1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca68c9a0c8fb4cccb081b404daff8b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:19.711452 1286 tablet_service.cc:1468] Processing CreateTablet for tablet ca68c9a0c8fb4cccb081b404daff8b2a (DEFAULT_TABLE table=default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:19.711935 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca68c9a0c8fb4cccb081b404daff8b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:19.712219 1597 sys_catalog.cc:263] Verifying existing consensus state
I20250901 01:53:19.714447 1422 tablet_bootstrap.cc:492] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:19.716176 1422 tablet_bootstrap.cc:654] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:19.718595 1597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: Bootstrap starting.
I20250901 01:53:19.719499 1424 tablet_bootstrap.cc:492] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:19.721213 1424 tablet_bootstrap.cc:654] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:19.724231 1423 tablet_bootstrap.cc:492] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:19.726661 1423 tablet_bootstrap.cc:654] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:19.733405 1597 log.cc:826] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: Log is configured to *not* fsync() on all Append() calls
I20250901 01:53:19.741521 1424 tablet_bootstrap.cc:492] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:19.741737 1424 ts_tablet_manager.cc:1397] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.010s
I20250901 01:53:19.742692 1424 raft_consensus.cc:357] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.742904 1424 raft_consensus.cc:383] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:19.743579 1424 raft_consensus.cc:738] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:19.743944 1424 consensus_queue.cc:260] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.746435 1424 ts_tablet_manager.cc:1428] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250901 01:53:19.753363 1423 tablet_bootstrap.cc:492] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:19.753664 1423 ts_tablet_manager.cc:1397] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.000s
I20250901 01:53:19.754828 1423 raft_consensus.cc:357] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.755091 1423 raft_consensus.cc:383] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:19.755224 1423 raft_consensus.cc:738] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:19.755491 1423 consensus_queue.cc:260] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.759896 1423 ts_tablet_manager.cc:1428] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250901 01:53:19.767124 1597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 01:53:19.774888 1597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: Bootstrap complete.
I20250901 01:53:19.776235 1450 raft_consensus.cc:491] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:19.777037 1450 raft_consensus.cc:513] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.777894 1450 leader_election.cc:290] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:19.778882 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e1a5e7ec3f842ffa6f24e5b315eb6dd" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:19.779136 1307 raft_consensus.cc:2466] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 0.
I20250901 01:53:19.779817 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e1a5e7ec3f842ffa6f24e5b315eb6dd" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:19.780062 1024 raft_consensus.cc:2466] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 0.
I20250901 01:53:19.780404 1478 raft_consensus.cc:491] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:19.780565 1478 raft_consensus.cc:513] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.781203 1478 leader_election.cc:290] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:19.782027 1101 leader_election.cc:304] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:19.783197 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca68c9a0c8fb4cccb081b404daff8b2a" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:19.783891 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca68c9a0c8fb4cccb081b404daff8b2a" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:19.784158 1307 raft_consensus.cc:2466] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:19.784703 954 leader_election.cc:304] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters:
W20250901 01:53:19.785356 953 leader_election.cc:343] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:53:19.785789 1478 raft_consensus.cc:2802] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:19.785881 1478 raft_consensus.cc:491] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:19.785974 1478 raft_consensus.cc:3058] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.788702 1450 raft_consensus.cc:2802] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:19.788837 1450 raft_consensus.cc:491] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:19.788967 1450 raft_consensus.cc:3058] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.791812 1450 raft_consensus.cc:513] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.793052 1422 tablet_bootstrap.cc:492] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:19.793291 1422 ts_tablet_manager.cc:1397] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.079s user 0.012s sys 0.019s
I20250901 01:53:19.793677 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e1a5e7ec3f842ffa6f24e5b315eb6dd" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:19.793915 1024 raft_consensus.cc:3058] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.794251 1422 raft_consensus.cc:357] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.794445 1422 raft_consensus.cc:383] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:19.794556 1422 raft_consensus.cc:738] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:19.794786 1422 consensus_queue.cc:260] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.795310 1422 ts_tablet_manager.cc:1428] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:53:19.797019 1024 raft_consensus.cc:2466] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 1.
I20250901 01:53:19.799274 1103 leader_election.cc:304] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:19.799620 1477 raft_consensus.cc:2802] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:19.800954 683 catalog_manager.cc:3460] renamed table default.table_6bcc2bc2c6574eca8387f75fc422beb9 [id=8ee4e0d7bca3410b9967df0dacf8614d] in HMS: new name default.table_5d5e7ea4ac7d4e5f9de3237058d11e80
I20250901 01:53:19.808452 1450 leader_election.cc:290] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Requested vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:19.809859 1307 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e1a5e7ec3f842ffa6f24e5b315eb6dd" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:19.810117 1307 raft_consensus.cc:3058] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.814419 1478 raft_consensus.cc:513] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.817366 1307 raft_consensus.cc:2466] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 1.
I20250901 01:53:19.819438 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca68c9a0c8fb4cccb081b404daff8b2a" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:19.819698 1167 raft_consensus.cc:3058] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.823259 1167 raft_consensus.cc:2466] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:19.823928 953 leader_election.cc:304] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:19.824331 1578 raft_consensus.cc:2802] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:19.824515 1578 raft_consensus.cc:695] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Becoming Leader. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Running, Role: LEADER
I20250901 01:53:19.824817 1578 consensus_queue.cc:237] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.813577 1597 raft_consensus.cc:357] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:19.825373 1597 raft_consensus.cc:738] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835a442fbc4c48eb9148ca677e410a59, State: Initialized, Role: FOLLOWER
I20250901 01:53:19.831449 1579 consensus_queue.cc:1035] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:19.839148 1579 consensus_queue.cc:1035] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:19.842224 1477 raft_consensus.cc:695] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [term 1 LEADER]: Becoming Leader. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Running, Role: LEADER
I20250901 01:53:19.842653 1477 consensus_queue.cc:237] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:19.844641 1478 leader_election.cc:290] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Requested vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:19.845508 1305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca68c9a0c8fb4cccb081b404daff8b2a" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:19.847085 1305 raft_consensus.cc:3058] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:19.851614 1305 raft_consensus.cc:2466] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:19.855286 1353 heartbeater.cc:344] Connected to a master server at 127.0.104.60:36985
I20250901 01:53:19.855593 1353 heartbeater.cc:499] Master 127.0.104.60:36985 was elected leader, sending a full tablet report...
I20250901 01:53:19.859946 1213 heartbeater.cc:344] Connected to a master server at 127.0.104.60:36985
I20250901 01:53:19.867187 1597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:19.876034 1478 consensus_queue.cc:1035] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:19.877699 1070 heartbeater.cc:344] Connected to a master server at 127.0.104.60:36985
I20250901 01:53:19.896451 1353 heartbeater.cc:499] Master 127.0.104.60:36985 was elected leader, sending a full tablet report...
I20250901 01:53:19.912276 1578 consensus_queue.cc:1035] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:19.920006 1353 heartbeater.cc:499] Master 127.0.104.60:36985 was elected leader, sending a full tablet report...
I20250901 01:53:19.925877 1615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:19.928457 1615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:19.930684 1597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 01:53:19.932546 1552 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 2 FOLLOWER]: Advancing to term 3
I20250901 01:53:19.937800 1353 heartbeater.cc:499] Master 127.0.104.60:36985 was elected leader, sending a full tablet report...
I20250901 01:53:19.995074 1615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c35f5a36eeb4a93aecdd71e2d1c3a21. Latest consensus state: current_term: 3 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:19.995394 1615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:19.996008 1622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:19.996420 1622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:19.998809 1477 consensus_queue.cc:1035] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:20.000284 1353 heartbeater.cc:499] Master 127.0.104.60:36985 was elected leader, sending a full tablet report...
I20250901 01:53:20.020529 1429 consensus_queue.cc:1035] T 5274573577374b47b99fa310a5ca83bb P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:20.027896 1353 heartbeater.cc:499] Master 127.0.104.60:36985 was elected leader, sending a full tablet report...
I20250901 01:53:20.078583 1620 mvcc.cc:204] Tried to move back new op lower bound from 7195408792368476160 to 7195408783233470464. Current Snapshot: MvccSnapshot[applied={T|T < 7195408792368476160 or (T in {7195408792368476160})}]
I20250901 01:53:20.131850 686 catalog_manager.cc:3460] renamed table default.table_476c1fc1206b4426855b5602d4b0cc88 [id=af5c975082de48cfa7295b2d8bd16635] in HMS: new name default.table_1ff7214e853748e58232b9bea0cb6912
I20250901 01:53:20.221902 1644 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:20.222230 1644 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:20.227912 699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b557230fc3b142c4bf5f6f58d241905f" candidate_term: 4 candidate_status { last_received { term: 3 index: 52 } } ignore_live_leader: false dest_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" is_pre_election: true
I20250901 01:53:20.228092 1644 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:20.239145 1353 heartbeater.cc:499] Master 127.0.104.60:36985 was elected leader, sending a full tablet report...
I20250901 01:53:20.245153 1640 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:20.245568 1640 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: loading cluster ID for follower catalog manager: success
I20250901 01:53:20.252440 1552 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b557230fc3b142c4bf5f6f58d241905f" candidate_term: 4 candidate_status { last_received { term: 3 index: 52 } } ignore_live_leader: false dest_uuid: "835a442fbc4c48eb9148ca677e410a59" is_pre_election: true
I20250901 01:53:20.253917 746 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:20.254583 1644 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250901 01:53:20.261752 1538 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ad2ba2450480454388723cd729cc438e" instance_seqno: 1756691596941203) as {username='slave'} at 127.0.104.3:44769; Asking this server to re-register.
I20250901 01:53:20.263161 1353 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:20.263128 684 catalog_manager.cc:2149] created HMS catalog entry for table [id=7185f28e50df4fb9a4bf41cc5b73169d]
I20250901 01:53:20.265776 1538 ts_manager.cc:194] Registered new tserver with Master: ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:20.275784 1640 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: acquiring CA information for follower catalog manager: success
I20250901 01:53:20.281627 1640 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 01:53:20.293812 1577 consensus_queue.cc:1035] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:20.317540 1538 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" instance_seqno: 1756691596380664) as {username='slave'} at 127.0.104.2:47559; Asking this server to re-register.
I20250901 01:53:20.318766 1213 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:20.321641 1538 ts_manager.cc:194] Registered new tserver with Master: e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:20.325201 1579 consensus_queue.cc:1035] T 8dd9f23bffa543a490c018108ad41051 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:20.326283 1538 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" instance_seqno: 1756691595802033) as {username='slave'} at 127.0.104.1:57663; Asking this server to re-register.
I20250901 01:53:20.327550 1070 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:20.329756 1538 ts_manager.cc:194] Registered new tserver with Master: b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:20.344452 1649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:20.344852 1649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:20.360986 684 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58182 (request call id 43) took 2117 ms. Trace:
I20250901 01:53:20.361213 684 rpcz_store.cc:276] 0901 01:53:18.243165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:18.665527 (+422362us) service_pool.cc:225] Handling call
0901 01:53:18.998765 (+333238us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:53:18.999082 (+ 317us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:53:20.263084 (+1264002us) catalog_manager.cc:2148] Created new table in HMS catalog
0901 01:53:20.360523 (+ 97439us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:53:20.360705 (+ 182us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:53:20.360885 (+ 180us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:53:20.282517 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:53:20.282786 (+ 269us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:53:20.282795 (+ 9us) write_op.cc:435] Acquired schema lock
0901 01:53:20.282798 (+ 3us) tablet.cc:660] Decoding operations
0901 01:53:20.282985 (+ 187us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:53:20.283002 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0901 01:53:20.283008 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:53:20.283136 (+ 128us) tablet.cc:699] Row locks acquired
0901 01:53:20.283139 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:53:20.283198 (+ 59us) write_op.cc:270] Start()
0901 01:53:20.283245 (+ 47us) write_op.cc:276] Timestamp: P: 1756691600283162 usec, L: 0
0901 01:53:20.283249 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:53:20.283581 (+ 332us) log.cc:844] Serialized 432 byte log entry
0901 01:53:20.344207 (+ 60626us) spinlock_profiling.cc:243] Waited 36.1 ms on lock 0x616000041eb8. stack: 00007fce7a8167d6 00007fce7a816590 00007fce798093ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fce861d12f9 00007fce861d66e4 00007fce861d64a8 0000000000a774dc 00007fce7a85cb5f 00007fce7a85fe2b 00007fce7a85fca8 0000000000a774dc 00007fce7a83bcc8 00007fce7dc5d6da
0901 01:53:20.345766 (+ 1559us) op_driver.cc:464] REPLICATION: finished
0901 01:53:20.358697 (+ 12931us) write_op.cc:301] APPLY: starting
0901 01:53:20.358733 (+ 36us) tablet.cc:1365] starting BulkCheckPresence
0901 01:53:20.358820 (+ 87us) tablet.cc:1368] finished BulkCheckPresence
0901 01:53:20.358825 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:53:20.359053 (+ 228us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:53:20.359251 (+ 198us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:53:20.359272 (+ 21us) write_op.cc:312] APPLY: finished
0901 01:53:20.359518 (+ 246us) log.cc:844] Serialized 28 byte log entry
0901 01:53:20.359963 (+ 445us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:20.360006 (+ 43us) write_op.cc:454] Released schema lock
0901 01:53:20.360020 (+ 14us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":1132262,"Hive Metastore.run_cpu_time_us":351,"Hive Metastore.run_wall_time_us":131284,"child_traces":[["op",{"apply.queue_time_us":12758,"leader-initialization.queue_time_us":552,"leader-initialization.run_cpu_time_us":21,"leader-initialization.run_wall_time_us":22,"mutex_wait_us":214,"num_ops":2,"prepare.queue_time_us":18590,"prepare.run_cpu_time_us":53391,"prepare.run_wall_time_us":63977,"raft.queue_time_us":63551,"raft.run_cpu_time_us":16291,"raft.run_wall_time_us":92390,"replication_time_us":62453,"spinlock_wait_cycles":83143936,"thread_start_us":127846,"threads_started":5}]]}
I20250901 01:53:20.383147 1146 tablet_service.cc:1468] Processing CreateTablet for tablet d258ddb89ab54c838f2fc8a08ab0f1a5 (DEFAULT_TABLE table=default.table_da487bf0d57f439bb9da2ec57880a73b [id=7185f28e50df4fb9a4bf41cc5b73169d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:20.383685 1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d258ddb89ab54c838f2fc8a08ab0f1a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:20.386585 1286 tablet_service.cc:1468] Processing CreateTablet for tablet d258ddb89ab54c838f2fc8a08ab0f1a5 (DEFAULT_TABLE table=default.table_da487bf0d57f439bb9da2ec57880a73b [id=7185f28e50df4fb9a4bf41cc5b73169d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:20.386693 1004 tablet_service.cc:1468] Processing CreateTablet for tablet d258ddb89ab54c838f2fc8a08ab0f1a5 (DEFAULT_TABLE table=default.table_da487bf0d57f439bb9da2ec57880a73b [id=7185f28e50df4fb9a4bf41cc5b73169d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:20.387068 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d258ddb89ab54c838f2fc8a08ab0f1a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:20.387084 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d258ddb89ab54c838f2fc8a08ab0f1a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:20.388036 684 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_56fde480a00a4e62b84eedba22b0bade"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:20.403210 1658 tablet_bootstrap.cc:492] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:20.406016 1658 tablet_bootstrap.cc:654] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:20.421849 1660 tablet_bootstrap.cc:492] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:20.429023 1658 tablet_bootstrap.cc:492] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:20.429390 1658 ts_tablet_manager.cc:1397] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
I20250901 01:53:20.430234 1659 tablet_bootstrap.cc:492] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:20.430442 1658 raft_consensus.cc:357] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:20.430756 1658 raft_consensus.cc:383] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:20.430895 1658 raft_consensus.cc:738] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:20.431196 1658 consensus_queue.cc:260] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:20.432368 1659 tablet_bootstrap.cc:654] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:20.432703 1658 ts_tablet_manager.cc:1428] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:53:20.436139 1660 tablet_bootstrap.cc:654] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:20.451900 1659 tablet_bootstrap.cc:492] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:20.452236 1659 ts_tablet_manager.cc:1397] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.007s
I20250901 01:53:20.456914 1659 raft_consensus.cc:357] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:20.457170 1659 raft_consensus.cc:383] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:20.457273 1659 raft_consensus.cc:738] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:20.457533 1659 consensus_queue.cc:260] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:20.458122 1659 ts_tablet_manager.cc:1428] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:53:20.463788 1660 tablet_bootstrap.cc:492] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:20.464080 1660 ts_tablet_manager.cc:1397] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.000s
I20250901 01:53:20.465103 1660 raft_consensus.cc:357] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:20.465400 1660 raft_consensus.cc:383] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:20.465534 1660 raft_consensus.cc:738] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:20.465832 1660 consensus_queue.cc:260] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:20.469532 1660 ts_tablet_manager.cc:1428] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250901 01:53:20.479039 1577 consensus_queue.cc:1035] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:20.499583 1578 consensus_queue.cc:1035] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:20.517587 1429 consensus_queue.cc:1035] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:20.519542 1647 raft_consensus.cc:491] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:20.519729 1647 raft_consensus.cc:513] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:20.520501 1647 leader_election.cc:290] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:20.522264 1166 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d258ddb89ab54c838f2fc8a08ab0f1a5" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:20.522553 1166 raft_consensus.cc:2466] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:20.522567 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d258ddb89ab54c838f2fc8a08ab0f1a5" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:20.522778 1024 raft_consensus.cc:2466] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:20.523254 1240 leader_election.cc:304] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:20.523622 1561 raft_consensus.cc:2802] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:20.523748 1561 raft_consensus.cc:491] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:20.523842 1561 raft_consensus.cc:3058] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:20.528138 1561 raft_consensus.cc:513] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:20.529628 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d258ddb89ab54c838f2fc8a08ab0f1a5" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:20.529628 1166 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d258ddb89ab54c838f2fc8a08ab0f1a5" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:20.529839 1024 raft_consensus.cc:3058] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:20.529840 1166 raft_consensus.cc:3058] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:20.532774 1166 raft_consensus.cc:2466] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:20.533207 1024 raft_consensus.cc:2466] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:20.533430 1561 leader_election.cc:290] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Requested vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:20.533769 1243 leader_election.cc:304] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:20.534088 1561 raft_consensus.cc:2802] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:20.534473 1561 raft_consensus.cc:695] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [term 1 LEADER]: Becoming Leader. State: Replica: ad2ba2450480454388723cd729cc438e, State: Running, Role: LEADER
I20250901 01:53:20.534919 1561 consensus_queue.cc:237] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:20.544147 1429 consensus_queue.cc:1035] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:20.818531 1429 consensus_queue.cc:1035] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:20.853912 1669 consensus_queue.cc:1035] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:20.888509 1674 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:20.888743 1674 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:20.890071 699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b557230fc3b142c4bf5f6f58d241905f" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" is_pre_election: true
I20250901 01:53:20.890362 1552 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b557230fc3b142c4bf5f6f58d241905f" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "835a442fbc4c48eb9148ca677e410a59" is_pre_election: true
I20250901 01:53:20.890700 1552 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b557230fc3b142c4bf5f6f58d241905f in term 3.
I20250901 01:53:20.891191 1674 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:20.891220 746 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835a442fbc4c48eb9148ca677e410a59, b557230fc3b142c4bf5f6f58d241905f; no voters:
I20250901 01:53:20.891530 1674 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250901 01:53:20.891656 1674 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:20.891788 1674 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 3 FOLLOWER]: Advancing to term 4
I20250901 01:53:20.896319 1674 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:20.900995 699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b557230fc3b142c4bf5f6f58d241905f" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21"
I20250901 01:53:20.902051 1552 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b557230fc3b142c4bf5f6f58d241905f" candidate_term: 4 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "835a442fbc4c48eb9148ca677e410a59"
I20250901 01:53:20.902420 1552 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 3 FOLLOWER]: Advancing to term 4
I20250901 01:53:20.906287 1674 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 4 election: Requested vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:20.906522 1552 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b557230fc3b142c4bf5f6f58d241905f in term 4.
I20250901 01:53:20.907186 746 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835a442fbc4c48eb9148ca677e410a59, b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21
I20250901 01:53:20.908727 1674 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 4 FOLLOWER]: Leader election won for term 4
I20250901 01:53:20.914783 1674 raft_consensus.cc:695] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 4 LEADER]: Becoming Leader. State: Replica: b557230fc3b142c4bf5f6f58d241905f, State: Running, Role: LEADER
I20250901 01:53:20.915424 1674 consensus_queue.cc:237] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 3.54, Last appended by leader: 54, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:20.918596 1675 raft_consensus.cc:491] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:20.920819 1675 raft_consensus.cc:513] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:20.924469 699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "835a442fbc4c48eb9148ca677e410a59" candidate_term: 5 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" is_pre_election: true
I20250901 01:53:20.924464 1676 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b557230fc3b142c4bf5f6f58d241905f. Latest consensus state: current_term: 4 leader_uuid: "b557230fc3b142c4bf5f6f58d241905f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:20.924760 1676 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:20.934978 1677 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 01:53:20.937095 1675 leader_election.cc:290] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025)
I20250901 01:53:20.945453 1677 catalog_manager.cc:671] Loaded metadata for table default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19]
I20250901 01:53:20.946334 1677 catalog_manager.cc:671] Loaded metadata for table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]
I20250901 01:53:20.946991 1677 catalog_manager.cc:671] Loaded metadata for table default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8]
I20250901 01:53:20.947472 1677 catalog_manager.cc:671] Loaded metadata for table default.table_795876dee30e438da68abc8a5bbae13f [id=61c047af6fff411f908946c6206ba414]
I20250901 01:53:20.947991 1677 catalog_manager.cc:671] Loaded metadata for table default.table_da487bf0d57f439bb9da2ec57880a73b [id=7185f28e50df4fb9a4bf41cc5b73169d]
I20250901 01:53:20.948515 1677 catalog_manager.cc:671] Loaded metadata for table default.table_6bcc2bc2c6574eca8387f75fc422beb9 [id=8ee4e0d7bca3410b9967df0dacf8614d]
I20250901 01:53:20.949118 1677 catalog_manager.cc:671] Loaded metadata for table default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98]
I20250901 01:53:20.949597 1677 catalog_manager.cc:671] Loaded metadata for table default.table_476c1fc1206b4426855b5602d4b0cc88 [id=af5c975082de48cfa7295b2d8bd16635]
I20250901 01:53:20.950177 1677 catalog_manager.cc:671] Loaded metadata for table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647]
I20250901 01:53:20.950729 1677 catalog_manager.cc:671] Loaded metadata for table default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9]
I20250901 01:53:20.951856 1677 catalog_manager.cc:671] Loaded metadata for table default.table_c95fbf8fa7a64aa18002f1e268cdea68 [id=d830537abb3c488ca7b925a8a491432a]
I20250901 01:53:20.952330 1677 catalog_manager.cc:671] Loaded metadata for table default.table_832e2227a6314ac2a050f76013e116fc [id=ed41948c8dd04b93b7d5fc44cede27f6]
I20250901 01:53:20.952826 1677 catalog_manager.cc:671] Loaded metadata for table default.table_e628b8e1d2214e458c8a283659d45a07 [id=f1db819ca90642eaac30e77fd5f5c8d5]
I20250901 01:53:20.955780 1677 tablet_loader.cc:96] loaded metadata for tablet 138d19e14b734c1cace0d7ba51ab2caa (table default.table_6bcc2bc2c6574eca8387f75fc422beb9 [id=8ee4e0d7bca3410b9967df0dacf8614d])
I20250901 01:53:20.956336 1677 tablet_loader.cc:96] loaded metadata for tablet 4e1a5e7ec3f842ffa6f24e5b315eb6dd (table default.table_c95fbf8fa7a64aa18002f1e268cdea68 [id=d830537abb3c488ca7b925a8a491432a])
I20250901 01:53:20.956785 1677 tablet_loader.cc:96] loaded metadata for tablet 5274573577374b47b99fa310a5ca83bb (table default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19])
I20250901 01:53:20.957218 1677 tablet_loader.cc:96] loaded metadata for tablet 586fbc1ef054435aaaea9ae961f15115 (table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647])
I20250901 01:53:20.957901 1677 tablet_loader.cc:96] loaded metadata for tablet 8d18463c27854cd28fa9de30c4990749 (table default.table_795876dee30e438da68abc8a5bbae13f [id=61c047af6fff411f908946c6206ba414])
I20250901 01:53:20.958403 1677 tablet_loader.cc:96] loaded metadata for tablet 8dd9f23bffa543a490c018108ad41051 (table default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98])
I20250901 01:53:20.958823 1677 tablet_loader.cc:96] loaded metadata for tablet 8f913036f34345fb8b20b3c9f431979e (table default.table_90a9fd7ee57d49b9ae8bb16bbbe41f50 [id=39123cfe4f62420681f8d679d2d376b8])
I20250901 01:53:20.959252 1677 tablet_loader.cc:96] loaded metadata for tablet 9cd7993f6a99468caa9b13fc4e8c0ba1 (table default.table_e628b8e1d2214e458c8a283659d45a07 [id=f1db819ca90642eaac30e77fd5f5c8d5])
I20250901 01:53:20.960166 1677 tablet_loader.cc:96] loaded metadata for tablet a6af5a07f3ca4fe4a8808b913551c091 (table default.table_476c1fc1206b4426855b5602d4b0cc88 [id=af5c975082de48cfa7295b2d8bd16635])
I20250901 01:53:20.960563 1677 tablet_loader.cc:96] loaded metadata for tablet a774cb6ccef24d958ba6c898c5e4786c (table default.table_832e2227a6314ac2a050f76013e116fc [id=ed41948c8dd04b93b7d5fc44cede27f6])
I20250901 01:53:20.960978 1677 tablet_loader.cc:96] loaded metadata for tablet b56b8dbd4e5b4892aa3b50409fc4584c (table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd])
I20250901 01:53:20.961668 1677 tablet_loader.cc:96] loaded metadata for tablet ca68c9a0c8fb4cccb081b404daff8b2a (table default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9])
I20250901 01:53:20.962442 1677 tablet_loader.cc:96] loaded metadata for tablet d258ddb89ab54c838f2fc8a08ab0f1a5 (table default.table_da487bf0d57f439bb9da2ec57880a73b [id=7185f28e50df4fb9a4bf41cc5b73169d])
I20250901 01:53:20.962810 1677 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 01:53:20.964660 1677 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:20.964756 1677 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 01:53:20.967433 1677 catalog_manager.cc:1506] Loading token signing keys...
I20250901 01:53:20.969013 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "835a442fbc4c48eb9148ca677e410a59" candidate_term: 5 candidate_status { last_received { term: 3 index: 54 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f" is_pre_election: true
I20250901 01:53:20.969791 1525 leader_election.cc:304] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835a442fbc4c48eb9148ca677e410a59; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, b557230fc3b142c4bf5f6f58d241905f
I20250901 01:53:20.970806 1675 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250901 01:53:20.972208 1677 catalog_manager.cc:5966] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Loaded TSK: 0
I20250901 01:53:20.972333 768 raft_consensus.cc:1238] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 4 LEADER]: Rejecting Update request from peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 for earlier term 3. Current term is 4. Ops: []
I20250901 01:53:20.973066 1680 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 }, Status: INVALID_TERM, Last received: 3.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250901 01:53:20.973539 1677 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 01:53:20.975307 1677 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 01:53:20.977016 1677 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 11
I20250901 01:53:20.977778 1680 consensus_queue.cc:1230] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Peer b557230fc3b142c4bf5f6f58d241905f log is divergent from this leader: its last log entry 4.55 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 54
I20250901 01:53:20.978099 1681 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 3 LEADER]: Stepping down as leader of term 3
I20250901 01:53:20.978243 1681 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2c35f5a36eeb4a93aecdd71e2d1c3a21, State: Running, Role: LEADER
I20250901 01:53:20.978655 1681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 3.54, Last appended by leader: 54, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:20.979027 1681 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 3 FOLLOWER]: Advancing to term 4
I20250901 01:53:20.984308 685 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_795876dee30e438da68abc8a5bbae13f [id=61c047af6fff411f908946c6206ba414]
W20250901 01:53:21.010325 905 catalog_manager.cc:3897] Illegal state: an error occurred while updating the sys-catalog: replica 2c35f5a36eeb4a93aecdd71e2d1c3a21 is not leader of this config: current role FOLLOWER
W20250901 01:53:21.010766 905 hms_notification_log_listener.cc:330] Failed to handle Hive Metastore notification: 12 ALTER_TABLE default.table_7e006da3e6d24d309af545afeac11ca4: Illegal state: an error occurred while updating the sys-catalog: replica 2c35f5a36eeb4a93aecdd71e2d1c3a21 is not leader of this config: current role FOLLOWER
W20250901 01:53:21.011046 905 hms_notification_log_listener.cc:143] Hive Metastore notification log listener poll failed: Service unavailable: lost leadership while handling Hive Metastore notification log events: an error occurred while updating the sys-catalog: replica 2c35f5a36eeb4a93aecdd71e2d1c3a21 is not leader of this config: current role FOLLOWER
I20250901 01:53:21.011333 687 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58182 (request call id 48) took 2689 ms. Trace:
I20250901 01:53:21.011440 687 rpcz_store.cc:276] 0901 01:53:18.322114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:19.062305 (+740191us) service_pool.cc:225] Handling call
0901 01:53:21.011296 (+1948991us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250901 01:53:21.017231 1362 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.62:35805)
W20250901 01:53:21.017886 684 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_56fde480a00a4e62b84eedba22b0bade in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:53:21.018342 681 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:58182 (request call id 39) took 2897 ms. Trace:
W20250901 01:53:21.018594 689 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_98a9d1400149426bab73e31aea263f5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:53:21.023042 683 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:58182 (request call id 38) took 2983 ms. Trace:
I20250901 01:53:21.023201 685 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:58182 (request call id 47) took 2704 ms. Trace:
I20250901 01:53:21.023350 685 rpcz_store.cc:276] 0901 01:53:18.318699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:18.914887 (+596188us) service_pool.cc:225] Handling call
0901 01:53:19.021400 (+106513us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:53:21.023144 (+2001744us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":1241141,"Hive Metastore.run_cpu_time_us":361,"Hive Metastore.run_wall_time_us":721354}
I20250901 01:53:21.024049 686 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:58182 (request call id 42) took 2796 ms. Trace:
I20250901 01:53:21.024196 686 rpcz_store.cc:276] 0901 01:53:18.227484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:18.517128 (+289644us) service_pool.cc:225] Handling call
0901 01:53:18.998347 (+481219us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:53:21.018842 (+2020495us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:53:21.023667 (+ 4825us) catalog_manager.cc:3683] Apply alter schema
0901 01:53:21.023996 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":802301,"Hive Metastore.run_cpu_time_us":824,"Hive Metastore.run_wall_time_us":330600}
I20250901 01:53:21.025367 685 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58182 (request call id 53) took 2397 ms. Trace:
I20250901 01:53:21.025507 685 rpcz_store.cc:276] 0901 01:53:18.627662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.024980 (+2397318us) service_pool.cc:225] Handling call
0901 01:53:21.025317 (+ 337us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:53:21.026051 685 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:58182 (request call id 54) took 2366 ms. Trace:
I20250901 01:53:21.026211 685 rpcz_store.cc:276] 0901 01:53:18.659293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.025679 (+2366386us) service_pool.cc:225] Handling call
0901 01:53:21.026005 (+ 326us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:53:21.026465 685 master_service.cc:948] ReplaceTablet: received request to replace tablet b56b8dbd4e5b4892aa3b50409fc4584c from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:21.026841 685 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:58182 (request call id 55) took 2362 ms. Trace:
I20250901 01:53:21.026877 688 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:58182 (request call id 29) took 3310 ms. Trace:
I20250901 01:53:21.026996 688 rpcz_store.cc:276] 0901 01:53:17.716660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:17.716721 (+ 61us) service_pool.cc:225] Handling call
0901 01:53:18.994825 (+1278104us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:53:21.018667 (+2023842us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:53:21.026501 (+ 7834us) catalog_manager.cc:3683] Apply alter schema
0901 01:53:21.026802 (+ 301us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":209688,"Hive Metastore.run_cpu_time_us":1006,"Hive Metastore.run_wall_time_us":107865}
I20250901 01:53:21.027608 688 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58182 (request call id 56) took 2316 ms. Trace:
I20250901 01:53:21.027699 688 rpcz_store.cc:276] 0901 01:53:18.710986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.027249 (+2316263us) service_pool.cc:225] Handling call
0901 01:53:21.027530 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:53:21.027942 688 master_service.cc:948] ReplaceTablet: received request to replace tablet c6380a3fefba4c4caed049b434425733 from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:21.028218 688 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:58182 (request call id 57) took 2241 ms. Trace:
I20250901 01:53:21.028316 688 rpcz_store.cc:276] 0901 01:53:18.787152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.027862 (+2240710us) service_pool.cc:225] Handling call
0901 01:53:21.028186 (+ 324us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250901 01:53:21.028635 1362 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.62:35805)
I20250901 01:53:21.028779 688 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58182 (request call id 58) took 2168 ms. Trace:
I20250901 01:53:21.028913 688 rpcz_store.cc:276] 0901 01:53:18.859895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.028487 (+2168592us) service_pool.cc:225] Handling call
0901 01:53:21.028744 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:53:21.029404 688 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:58182 (request call id 59) took 2004 ms. Trace:
I20250901 01:53:21.029523 688 rpcz_store.cc:276] 0901 01:53:19.024993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.029105 (+2004112us) service_pool.cc:225] Handling call
0901 01:53:21.029366 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:53:21.029783 688 master_service.cc:948] ReplaceTablet: received request to replace tablet a774cb6ccef24d958ba6c898c5e4786c from {username='slave'} at 127.0.0.1:58182
W20250901 01:53:21.029954 1362 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.62:35805)
I20250901 01:53:21.023249 683 rpcz_store.cc:276] 0901 01:53:18.039917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:18.317952 (+278035us) service_pool.cc:225] Handling call
0901 01:53:18.998119 (+680167us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:53:21.018788 (+2020669us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:53:21.022535 (+ 3747us) catalog_manager.cc:3683] Apply alter schema
0901 01:53:21.022965 (+ 430us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":690079,"Hive Metastore.run_cpu_time_us":890,"Hive Metastore.run_wall_time_us":112300}
I20250901 01:53:21.031956 683 master_service.cc:948] ReplaceTablet: received request to replace tablet b56b8dbd4e5b4892aa3b50409fc4584c from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:21.032332 683 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:58182 (request call id 61) took 1908 ms. Trace:
I20250901 01:53:21.032449 683 rpcz_store.cc:276] 0901 01:53:19.123381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.031889 (+1908508us) service_pool.cc:225] Handling call
0901 01:53:21.032275 (+ 386us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:53:21.032712 683 master_service.cc:948] ReplaceTablet: received request to replace tablet b56b8dbd4e5b4892aa3b50409fc4584c from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:21.032997 683 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:58182 (request call id 62) took 1903 ms. Trace:
I20250901 01:53:21.033111 683 rpcz_store.cc:276] 0901 01:53:19.129188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.032628 (+1903440us) service_pool.cc:225] Handling call
0901 01:53:21.032955 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:53:21.033638 683 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:58182 (request call id 63) took 1612 ms. Trace:
I20250901 01:53:21.033766 683 rpcz_store.cc:276] 0901 01:53:19.421462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.033282 (+1611820us) service_pool.cc:225] Handling call
0901 01:53:21.033588 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250901 01:53:21.034420 1362 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.61:33025)
I20250901 01:53:21.039279 755 master_service.cc:948] ReplaceTablet: received request to replace tablet c6380a3fefba4c4caed049b434425733 from {username='slave'} at 127.0.0.1:55080
I20250901 01:53:21.043334 1578 consensus_queue.cc:1035] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:21.046229 1578 consensus_queue.cc:1035] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:21.018494 681 rpcz_store.cc:276] 0901 01:53:18.121310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:18.353702 (+232392us) service_pool.cc:225] Handling call
0901 01:53:18.994662 (+640960us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:53:21.017456 (+2022794us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:53:21.017896 (+ 440us) catalog_manager.cc:3683] Apply alter schema
0901 01:53:21.018283 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":10141,"Hive Metastore.run_cpu_time_us":641,"Hive Metastore.run_wall_time_us":199542}
I20250901 01:53:21.048544 681 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58182 (request call id 65) took 1297 ms. Trace:
I20250901 01:53:21.048686 681 rpcz_store.cc:276] 0901 01:53:19.751452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.048044 (+1296592us) service_pool.cc:225] Handling call
0901 01:53:21.048500 (+ 456us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:53:21.049274 681 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58182 (request call id 66) took 1150 ms. Trace:
I20250901 01:53:21.049402 681 rpcz_store.cc:276] 0901 01:53:19.899218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.048909 (+1149691us) service_pool.cc:225] Handling call
0901 01:53:21.049235 (+ 326us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:53:21.053011 755 master_service.cc:948] ReplaceTablet: received request to replace tablet b56b8dbd4e5b4892aa3b50409fc4584c from {username='slave'} at 127.0.0.1:55080
I20250901 01:53:21.054914 752 master_service.cc:948] ReplaceTablet: received request to replace tablet b56b8dbd4e5b4892aa3b50409fc4584c from {username='slave'} at 127.0.0.1:55080
I20250901 01:53:21.072901 686 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:58158 (request call id 2) took 1206 ms. Trace:
I20250901 01:53:21.038599 758 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55080:
table { table_name: "default.Table_7722e49197be4ca9b9c7e67052f07c4b" } modify_external_catalogs: true
I20250901 01:53:21.051786 751 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55080:
table { table_name: "default.Table_61dfd07a8f784854b5d9e104e010daaa" } new_table_name: "default.Table_757f6c2777014fe1b0686fc1405b4b5f" modify_external_catalogs: true
I20250901 01:53:21.088256 758 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55080:
table { table_name: "default.Table_7722e49197be4ca9b9c7e67052f07c4b" } modify_external_catalogs: true
I20250901 01:53:21.040752 753 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55080:
table { table_name: "default.Table_d1c05496169b472fa9b021013c4afd01" } modify_external_catalogs: true
I20250901 01:53:21.088729 753 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55080:
table { table_name: "default.Table_d1c05496169b472fa9b021013c4afd01" } modify_external_catalogs: true
I20250901 01:53:21.040459 754 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55080:
name: "default.Table_facefcd0049d44a2b465d8315527aa08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:21.038962 756 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55080:
name: "default.Table_8082166db25745e78737b640063b3219"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:21.064411 750 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55080:
name: "default.Table_bb66c7b604104254a48d435595b83fc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:21.063102 757 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55080:
name: "default.Table_9f3451ed25714464ad16b3e94214d0e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:21.097574 1552 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 4 FOLLOWER]: Refusing update from remote peer b557230fc3b142c4bf5f6f58d241905f: Log matching property violated. Preceding OpId in replica: term: 3 index: 54. Preceding OpId from leader: term: 4 index: 56. (index mismatch)
I20250901 01:53:21.098551 1676 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [LEADER]: Connected to new peer: Peer: permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250901 01:53:21.100854 686 rpcz_store.cc:276] 0901 01:53:19.866905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.072479 (+1205574us) service_pool.cc:225] Handling call
0901 01:53:21.072851 (+ 372us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:53:21.102308 699 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 4 FOLLOWER]: Refusing update from remote peer b557230fc3b142c4bf5f6f58d241905f: Log matching property violated. Preceding OpId in replica: term: 3 index: 54. Preceding OpId from leader: term: 4 index: 56. (index mismatch)
I20250901 01:53:21.103008 1676 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250901 01:53:21.073403 683 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58182 (request call id 64) took 1453 ms. Trace:
I20250901 01:53:21.105906 683 rpcz_store.cc:276] 0901 01:53:19.619751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.033969 (+1414218us) service_pool.cc:225] Handling call
0901 01:53:21.073361 (+ 39392us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:53:21.073933 681 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:58162 (request call id 5) took 1183 ms. Trace:
I20250901 01:53:21.106874 681 rpcz_store.cc:276] 0901 01:53:19.890099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.073569 (+1183470us) service_pool.cc:225] Handling call
0901 01:53:21.073893 (+ 324us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:53:21.026960 685 rpcz_store.cc:276] 0901 01:53:18.663881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.026388 (+2362507us) service_pool.cc:225] Handling call
0901 01:53:21.026770 (+ 382us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:53:21.081679 688 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:58182 (request call id 60) took 1908 ms. Trace:
I20250901 01:53:21.109572 1675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b557230fc3b142c4bf5f6f58d241905f. Latest consensus state: current_term: 4 leader_uuid: "b557230fc3b142c4bf5f6f58d241905f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:21.109877 1675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:21.109846 688 rpcz_store.cc:276] 0901 01:53:19.122172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.029726 (+1907554us) service_pool.cc:225] Handling call
0901 01:53:21.030113 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:53:21.036780 759 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55080:
name: "default.Table_99b272393a1b40b4ae2d3bbaedc9ea60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:21.133543 1675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "b557230fc3b142c4bf5f6f58d241905f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:21.133817 1675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:21.137763 1674 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "b557230fc3b142c4bf5f6f58d241905f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:21.138077 1674 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:21.140719 752 catalog_manager.cc:5582] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e reported cstate change: term changed from 0 to 1, leader changed from <none> to b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b65b3154aefd49f7a37f70e9207e9853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } }
I20250901 01:53:21.143558 755 catalog_manager.cc:5869] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Sending DeleteTablet for 3 replicas of tablet b56b8dbd4e5b4892aa3b50409fc4584c
I20250901 01:53:21.146711 755 catalog_manager.cc:6652] ReplaceTablet: tablet b56b8dbd4e5b4892aa3b50409fc4584c deleted and replaced by tablet f81df1b890684c9cb01597c5bfebcefa
I20250901 01:53:21.182127 1354 heartbeater.cc:499] Master 127.0.104.61:33025 was elected leader, sending a full tablet report...
I20250901 01:53:21.183220 752 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55080:
name: "default.Table_167322850c734684b896b875526a5c6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:21.185200 1071 heartbeater.cc:499] Master 127.0.104.61:33025 was elected leader, sending a full tablet report...
I20250901 01:53:21.187820 1681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b557230fc3b142c4bf5f6f58d241905f. Latest consensus state: current_term: 4 leader_uuid: "b557230fc3b142c4bf5f6f58d241905f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:21.188099 1681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:21.191762 1676 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "b557230fc3b142c4bf5f6f58d241905f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:21.192044 1676 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:21.192467 1004 tablet_service.cc:1515] Processing DeleteTablet for tablet b56b8dbd4e5b4892aa3b50409fc4584c with delete_type TABLET_DATA_DELETED (replaced by tablet f81df1b890684c9cb01597c5bfebcefa) from {username='slave'} at 127.0.0.1:41698
I20250901 01:53:21.193673 1146 tablet_service.cc:1515] Processing DeleteTablet for tablet b56b8dbd4e5b4892aa3b50409fc4584c with delete_type TABLET_DATA_DELETED (replaced by tablet f81df1b890684c9cb01597c5bfebcefa) from {username='slave'} at 127.0.0.1:57118
I20250901 01:53:21.202781 1286 tablet_service.cc:1515] Processing DeleteTablet for tablet b56b8dbd4e5b4892aa3b50409fc4584c with delete_type TABLET_DATA_DELETED (replaced by tablet f81df1b890684c9cb01597c5bfebcefa) from {username='slave'} at 127.0.0.1:53910
I20250901 01:53:21.204882 755 catalog_manager.cc:5674] TS ad2ba2450480454388723cd729cc438e (127.0.104.3:42927) does not have the latest schema for tablet 9cd7993f6a99468caa9b13fc4e8c0ba1 (table default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5]). Expected version 1 got 0
I20250901 01:53:21.206552 1214 heartbeater.cc:499] Master 127.0.104.61:33025 was elected leader, sending a full tablet report...
I20250901 01:53:21.206674 1147 tablet_service.cc:1468] Processing CreateTablet for tablet f81df1b890684c9cb01597c5bfebcefa (DEFAULT_TABLE table=default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:21.207165 1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f81df1b890684c9cb01597c5bfebcefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:21.209249 1002 tablet_service.cc:1468] Processing CreateTablet for tablet f81df1b890684c9cb01597c5bfebcefa (DEFAULT_TABLE table=default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:21.209782 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f81df1b890684c9cb01597c5bfebcefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:21.214771 1287 tablet_service.cc:1468] Processing CreateTablet for tablet f81df1b890684c9cb01597c5bfebcefa (DEFAULT_TABLE table=default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:21.215265 1287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f81df1b890684c9cb01597c5bfebcefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:21.214039 755 catalog_manager.cc:5582] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P ad2ba2450480454388723cd729cc438e reported cstate change: term changed from 0 to 1, leader changed from <none> to e3c774a4edf44d6098a2332e734967a5 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "e3c774a4edf44d6098a2332e734967a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } }
I20250901 01:53:21.221645 755 catalog_manager.cc:5582] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e reported cstate change: term changed from 0 to 1, leader changed from <none> to ad2ba2450480454388723cd729cc438e (127.0.104.3). New cstate: current_term: 1 leader_uuid: "ad2ba2450480454388723cd729cc438e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } health_report { overall_health: HEALTHY } } }
I20250901 01:53:21.224108 755 catalog_manager.cc:5674] TS ad2ba2450480454388723cd729cc438e (127.0.104.3:42927) does not have the latest schema for tablet 8f913036f34345fb8b20b3c9f431979e (table default.table_8f21a26e011c4ed1b1ae7cb364e09195 [id=39123cfe4f62420681f8d679d2d376b8]). Expected version 1 got 0
I20250901 01:53:21.229743 680 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_832e2227a6314ac2a050f76013e116fc [id=ed41948c8dd04b93b7d5fc44cede27f6]
I20250901 01:53:21.242254 1702 tablet_replica.cc:331] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853: stopping tablet replica
I20250901 01:53:21.242617 1702 raft_consensus.cc:2241] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:53:21.243175 1702 raft_consensus.cc:2270] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:21.244197 1702 ts_tablet_manager.cc:1905] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:21.246012 680 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:58182 (request call id 51) took 2735 ms. Trace:
I20250901 01:53:21.245949 1681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "b557230fc3b142c4bf5f6f58d241905f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:21.246251 1681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:21.246183 680 rpcz_store.cc:276] 0901 01:53:18.510850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:19.696805 (+1185955us) service_pool.cc:225] Handling call
0901 01:53:21.017452 (+1320647us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:53:21.245959 (+228507us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":48,"Hive Metastore.run_cpu_time_us":218,"Hive Metastore.run_wall_time_us":211690}
I20250901 01:53:21.247705 1705 tablet_replica.cc:331] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e: stopping tablet replica
I20250901 01:53:21.247954 1705 raft_consensus.cc:2241] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:21.249257 1703 tablet_replica.cc:331] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5: stopping tablet replica
I20250901 01:53:21.249449 1705 raft_consensus.cc:2270] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:21.262226 1705 ts_tablet_manager.cc:1905] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:21.262825 1703 raft_consensus.cc:2241] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:21.263550 1703 raft_consensus.cc:2270] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:21.270947 1710 tablet.cc:1722] T 8f913036f34345fb8b20b3c9f431979e P e3c774a4edf44d6098a2332e734967a5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:53:21.273350 1713 tablet_bootstrap.cc:492] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:21.272746 1703 ts_tablet_manager.cc:1905] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:21.298897 1719 tablet.cc:1722] T 8f913036f34345fb8b20b3c9f431979e P ad2ba2450480454388723cd729cc438e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:53:21.303997 1721 tablet.cc:1722] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:53:21.311067 1713 tablet_bootstrap.cc:654] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:21.310640 1720 tablet.cc:1722] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:53:21.328725 1705 ts_tablet_manager.cc:1918] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:21.329025 1705 log.cc:1199] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal/wals/b56b8dbd4e5b4892aa3b50409fc4584c
I20250901 01:53:21.330296 1705 ts_tablet_manager.cc:1939] T b56b8dbd4e5b4892aa3b50409fc4584c P ad2ba2450480454388723cd729cc438e: Deleting consensus metadata
I20250901 01:53:21.332803 1702 ts_tablet_manager.cc:1918] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:21.333094 1702 log.cc:1199] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/wals/b56b8dbd4e5b4892aa3b50409fc4584c
I20250901 01:53:21.333680 1702 ts_tablet_manager.cc:1939] T b56b8dbd4e5b4892aa3b50409fc4584c P b65b3154aefd49f7a37f70e9207e9853: Deleting consensus metadata
I20250901 01:53:21.335429 921 catalog_manager.cc:2747] Deleting table default.table_795876dee30e438da68abc8a5bbae13f [id=61c047af6fff411f908946c6206ba414] in response to Hive Metastore notification log event 20
I20250901 01:53:21.335398 1710 tablet.cc:1722] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:53:21.336922 1722 tablet_bootstrap.cc:492] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:21.339746 1722 tablet_bootstrap.cc:654] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:21.346063 746 catalog_manager.cc:4928] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): tablet b56b8dbd4e5b4892aa3b50409fc4584c (table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]) successfully deleted
I20250901 01:53:21.344653 1713 tablet_bootstrap.cc:492] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:21.346609 1713 ts_tablet_manager.cc:1397] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.073s user 0.006s sys 0.000s
I20250901 01:53:21.355882 755 catalog_manager.cc:5674] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315) does not have the latest schema for tablet 9cd7993f6a99468caa9b13fc4e8c0ba1 (table default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5]). Expected version 1 got 0
I20250901 01:53:21.356345 755 catalog_manager.cc:5674] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315) does not have the latest schema for tablet a6af5a07f3ca4fe4a8808b913551c091 (table default.table_1ff7214e853748e58232b9bea0cb6912 [id=af5c975082de48cfa7295b2d8bd16635]). Expected version 1 got 0
I20250901 01:53:21.356541 755 catalog_manager.cc:5674] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315) does not have the latest schema for tablet 138d19e14b734c1cace0d7ba51ab2caa (table default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d]). Expected version 1 got 0
I20250901 01:53:21.357007 755 catalog_manager.cc:5674] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315) does not have the latest schema for tablet 8f913036f34345fb8b20b3c9f431979e (table default.table_8f21a26e011c4ed1b1ae7cb364e09195 [id=39123cfe4f62420681f8d679d2d376b8]). Expected version 1 got 0
I20250901 01:53:21.357280 921 catalog_manager.cc:5869] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Sending DeleteTablet for 3 replicas of tablet 8d18463c27854cd28fa9de30c4990749
I20250901 01:53:21.359079 1002 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d18463c27854cd28fa9de30c4990749 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:53:21 UTC) from {username='slave'} at 127.0.0.1:41698
I20250901 01:53:21.369578 1713 raft_consensus.cc:357] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:21.369900 1713 raft_consensus.cc:383] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:21.369990 1713 raft_consensus.cc:738] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:21.355633 1703 ts_tablet_manager.cc:1918] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:21.370597 1146 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d18463c27854cd28fa9de30c4990749 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:53:21 UTC) from {username='slave'} at 127.0.0.1:57118
I20250901 01:53:21.371513 1713 consensus_queue.cc:260] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:21.371878 1703 log.cc:1199] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal/wals/b56b8dbd4e5b4892aa3b50409fc4584c
I20250901 01:53:21.341342 1724 tablet_bootstrap.cc:492] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:21.359915 1286 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d18463c27854cd28fa9de30c4990749 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:53:21 UTC) from {username='slave'} at 127.0.0.1:53910
I20250901 01:53:21.372721 744 catalog_manager.cc:4928] TS ad2ba2450480454388723cd729cc438e (127.0.104.3:42927): tablet b56b8dbd4e5b4892aa3b50409fc4584c (table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]) successfully deleted
I20250901 01:53:21.374583 1713 ts_tablet_manager.cc:1428] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.027s user 0.003s sys 0.000s
I20250901 01:53:21.374802 1721 tablet.cc:1722] T a6af5a07f3ca4fe4a8808b913551c091 P e3c774a4edf44d6098a2332e734967a5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:53:21.374904 1002 tablet_service.cc:1515] Processing DeleteTablet for tablet b56b8dbd4e5b4892aa3b50409fc4584c with delete_type TABLET_DATA_DELETED (replaced by tablet f81df1b890684c9cb01597c5bfebcefa) from {username='slave'} at 127.0.0.1:41698
I20250901 01:53:21.375228 1702 tablet_replica.cc:331] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853: stopping tablet replica
I20250901 01:53:21.375936 1702 raft_consensus.cc:2241] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:53:21.377605 1724 tablet_bootstrap.cc:654] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:21.378597 1702 raft_consensus.cc:2270] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:21.381240 1703 ts_tablet_manager.cc:1939] T b56b8dbd4e5b4892aa3b50409fc4584c P e3c774a4edf44d6098a2332e734967a5: Deleting consensus metadata
I20250901 01:53:21.377840 1705 tablet_replica.cc:331] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e: stopping tablet replica
I20250901 01:53:21.381603 1705 raft_consensus.cc:2241] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:21.381848 1705 raft_consensus.cc:2270] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:21.382476 1703 tablet_replica.cc:331] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5: stopping tablet replica
I20250901 01:53:21.382714 1703 raft_consensus.cc:2241] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:21.382930 1703 raft_consensus.cc:2270] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:21.383545 1703 ts_tablet_manager.cc:1905] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:21.384298 1730 tablet.cc:1722] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:53:21.384616 1702 ts_tablet_manager.cc:1905] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:21.384769 1705 ts_tablet_manager.cc:1905] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:21.385238 743 catalog_manager.cc:4928] TS e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245): tablet b56b8dbd4e5b4892aa3b50409fc4584c (table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]) successfully deleted
I20250901 01:53:21.389860 1709 tablet.cc:1722] T 8f913036f34345fb8b20b3c9f431979e P b65b3154aefd49f7a37f70e9207e9853: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:53:21.395802 755 catalog_manager.cc:5674] TS e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245) does not have the latest schema for tablet 9cd7993f6a99468caa9b13fc4e8c0ba1 (table default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5]). Expected version 1 got 0
I20250901 01:53:21.396176 755 catalog_manager.cc:5674] TS e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245) does not have the latest schema for tablet a6af5a07f3ca4fe4a8808b913551c091 (table default.table_1ff7214e853748e58232b9bea0cb6912 [id=af5c975082de48cfa7295b2d8bd16635]). Expected version 1 got 0
I20250901 01:53:21.396451 755 catalog_manager.cc:5674] TS e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245) does not have the latest schema for tablet 138d19e14b734c1cace0d7ba51ab2caa (table default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d]). Expected version 1 got 0
I20250901 01:53:21.397070 755 catalog_manager.cc:5674] TS e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245) does not have the latest schema for tablet 8f913036f34345fb8b20b3c9f431979e (table default.table_8f21a26e011c4ed1b1ae7cb364e09195 [id=39123cfe4f62420681f8d679d2d376b8]). Expected version 1 got 0
I20250901 01:53:21.401914 1719 tablet.cc:1722] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:53:21.387822 1720 tablet.cc:1722] T a6af5a07f3ca4fe4a8808b913551c091 P ad2ba2450480454388723cd729cc438e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:53:21.406246 1725 tablet.cc:1722] T a6af5a07f3ca4fe4a8808b913551c091 P b65b3154aefd49f7a37f70e9207e9853: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:53:21.411844 921 catalog_manager.cc:2747] Deleting table default.table_832e2227a6314ac2a050f76013e116fc [id=ed41948c8dd04b93b7d5fc44cede27f6] in response to Hive Metastore notification log event 21
I20250901 01:53:21.413025 684 catalog_manager.cc:2149] created HMS catalog entry for table [id=181eeac3e2394bcbaf3071e3fad223d8]
W20250901 01:53:21.413942 684 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 2c35f5a36eeb4a93aecdd71e2d1c3a21 is not leader of this config: current role FOLLOWER
I20250901 01:53:21.416419 1724 tablet_bootstrap.cc:492] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:21.416822 1733 tablet.cc:1722] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:53:21.417637 1724 ts_tablet_manager.cc:1397] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.076s user 0.000s sys 0.005s
I20250901 01:53:21.422060 1724 raft_consensus.cc:357] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:21.422279 921 catalog_manager.cc:5869] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Sending DeleteTablet for 3 replicas of tablet a774cb6ccef24d958ba6c898c5e4786c
I20250901 01:53:21.424424 1002 tablet_service.cc:1515] Processing DeleteTablet for tablet a774cb6ccef24d958ba6c898c5e4786c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:53:21 UTC) from {username='slave'} at 127.0.0.1:41698
I20250901 01:53:21.431252 1722 tablet_bootstrap.cc:492] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:21.431447 1722 ts_tablet_manager.cc:1397] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.095s user 0.006s sys 0.000s
W20250901 01:53:21.431517 754 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_facefcd0049d44a2b465d8315527aa08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:53:21.432323 1722 raft_consensus.cc:357] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:21.432516 1147 tablet_service.cc:1515] Processing DeleteTablet for tablet a774cb6ccef24d958ba6c898c5e4786c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:53:21 UTC) from {username='slave'} at 127.0.0.1:57118
W20250901 01:53:21.433419 756 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8082166db25745e78737b640063b3219 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:53:21.432575 1722 raft_consensus.cc:383] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:21.433727 1722 raft_consensus.cc:738] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:21.434026 1722 consensus_queue.cc:260] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
W20250901 01:53:21.434531 750 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_bb66c7b604104254a48d435595b83fc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:53:21.435302 1722 ts_tablet_manager.cc:1428] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20250901 01:53:21.435628 757 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9f3451ed25714464ad16b3e94214d0e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:53:21.436705 759 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_99b272393a1b40b4ae2d3bbaedc9ea60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:53:21.436934 1724 raft_consensus.cc:383] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:21.437067 1724 raft_consensus.cc:738] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:21.437388 1724 consensus_queue.cc:260] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
W20250901 01:53:21.437819 752 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_167322850c734684b896b875526a5c6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:53:21.439652 1724 ts_tablet_manager.cc:1428] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250901 01:53:21.437387 1705 ts_tablet_manager.cc:1918] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:21.440392 1705 log.cc:1199] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal/wals/8d18463c27854cd28fa9de30c4990749
I20250901 01:53:21.441031 1287 tablet_service.cc:1515] Processing DeleteTablet for tablet a774cb6ccef24d958ba6c898c5e4786c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:53:21 UTC) from {username='slave'} at 127.0.0.1:53910
I20250901 01:53:21.441694 1705 ts_tablet_manager.cc:1939] T 8d18463c27854cd28fa9de30c4990749 P ad2ba2450480454388723cd729cc438e: Deleting consensus metadata
I20250901 01:53:21.442991 744 catalog_manager.cc:4928] TS ad2ba2450480454388723cd729cc438e (127.0.104.3:42927): tablet 8d18463c27854cd28fa9de30c4990749 (table default.table_795876dee30e438da68abc8a5bbae13f [id=61c047af6fff411f908946c6206ba414]) successfully deleted
I20250901 01:53:21.443384 1705 tablet_replica.cc:331] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e: stopping tablet replica
I20250901 01:53:21.443593 1705 raft_consensus.cc:2241] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:21.443828 1705 raft_consensus.cc:2270] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:21.447392 1705 ts_tablet_manager.cc:1905] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:21.451100 1147 tablet_service.cc:1515] Processing DeleteTablet for tablet b56b8dbd4e5b4892aa3b50409fc4584c with delete_type TABLET_DATA_DELETED (replaced by tablet f81df1b890684c9cb01597c5bfebcefa) from {username='slave'} at 127.0.0.1:57118
I20250901 01:53:21.461617 1702 ts_tablet_manager.cc:1918] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:21.461817 1702 log.cc:1199] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/wals/8d18463c27854cd28fa9de30c4990749
I20250901 01:53:21.462306 1702 ts_tablet_manager.cc:1939] T 8d18463c27854cd28fa9de30c4990749 P b65b3154aefd49f7a37f70e9207e9853: Deleting consensus metadata
I20250901 01:53:21.463557 746 catalog_manager.cc:4928] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): tablet 8d18463c27854cd28fa9de30c4990749 (table default.table_795876dee30e438da68abc8a5bbae13f [id=61c047af6fff411f908946c6206ba414]) successfully deleted
I20250901 01:53:21.479009 1703 ts_tablet_manager.cc:1918] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:21.479311 1703 log.cc:1199] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal/wals/8d18463c27854cd28fa9de30c4990749
I20250901 01:53:21.479852 1703 ts_tablet_manager.cc:1939] T 8d18463c27854cd28fa9de30c4990749 P e3c774a4edf44d6098a2332e734967a5: Deleting consensus metadata
I20250901 01:53:21.482051 755 catalog_manager.cc:6127] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: default.table_8f21a26e011c4ed1b1ae7cb364e09195 [id=39123cfe4f62420681f8d679d2d376b8] alter complete (version 1)
I20250901 01:53:21.486129 743 catalog_manager.cc:4928] TS e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245): tablet 8d18463c27854cd28fa9de30c4990749 (table default.table_795876dee30e438da68abc8a5bbae13f [id=61c047af6fff411f908946c6206ba414]) successfully deleted
I20250901 01:53:21.486668 1703 tablet_replica.cc:331] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5: stopping tablet replica
I20250901 01:53:21.486948 1703 raft_consensus.cc:2241] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:21.487233 1703 raft_consensus.cc:2270] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:21.487874 1703 ts_tablet_manager.cc:1905] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:21.487823 755 catalog_manager.cc:5674] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315) does not have the latest schema for tablet 8f913036f34345fb8b20b3c9f431979e (table default.table_8f21a26e011c4ed1b1ae7cb364e09195 [id=39123cfe4f62420681f8d679d2d376b8]). Expected version 1 got 0
I20250901 01:53:21.492537 1702 tablet_replica.cc:331] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853: stopping tablet replica
I20250901 01:53:21.492798 1702 raft_consensus.cc:2241] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Raft consensus shutting down.
W20250901 01:53:21.492905 746 catalog_manager.cc:4895] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): delete failed for tablet b56b8dbd4e5b4892aa3b50409fc4584c because the tablet was not found. No further retry: Not found: Tablet not found: b56b8dbd4e5b4892aa3b50409fc4584c
I20250901 01:53:21.493212 1702 raft_consensus.cc:2270] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:21.493860 1702 ts_tablet_manager.cc:1905] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:21.503722 1705 ts_tablet_manager.cc:1918] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:21.503897 1705 log.cc:1199] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal/wals/a774cb6ccef24d958ba6c898c5e4786c
I20250901 01:53:21.504341 1705 ts_tablet_manager.cc:1939] T a774cb6ccef24d958ba6c898c5e4786c P ad2ba2450480454388723cd729cc438e: Deleting consensus metadata
I20250901 01:53:21.505610 744 catalog_manager.cc:4928] TS ad2ba2450480454388723cd729cc438e (127.0.104.3:42927): tablet a774cb6ccef24d958ba6c898c5e4786c (table default.table_832e2227a6314ac2a050f76013e116fc [id=ed41948c8dd04b93b7d5fc44cede27f6]) successfully deleted
I20250901 01:53:21.507256 755 catalog_manager.cc:6127] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d] alter complete (version 1)
I20250901 01:53:21.508352 1702 ts_tablet_manager.cc:1918] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:21.508500 1702 log.cc:1199] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/wals/a774cb6ccef24d958ba6c898c5e4786c
I20250901 01:53:21.508948 1702 ts_tablet_manager.cc:1939] T a774cb6ccef24d958ba6c898c5e4786c P b65b3154aefd49f7a37f70e9207e9853: Deleting consensus metadata
I20250901 01:53:21.511173 1703 ts_tablet_manager.cc:1918] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:21.511369 1703 log.cc:1199] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal/wals/a774cb6ccef24d958ba6c898c5e4786c
I20250901 01:53:21.511852 1703 ts_tablet_manager.cc:1939] T a774cb6ccef24d958ba6c898c5e4786c P e3c774a4edf44d6098a2332e734967a5: Deleting consensus metadata
I20250901 01:53:21.512521 746 catalog_manager.cc:4928] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): tablet a774cb6ccef24d958ba6c898c5e4786c (table default.table_832e2227a6314ac2a050f76013e116fc [id=ed41948c8dd04b93b7d5fc44cede27f6]) successfully deleted
I20250901 01:53:21.514705 743 catalog_manager.cc:4928] TS e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245): tablet a774cb6ccef24d958ba6c898c5e4786c (table default.table_832e2227a6314ac2a050f76013e116fc [id=ed41948c8dd04b93b7d5fc44cede27f6]) successfully deleted
W20250901 01:53:21.515079 743 catalog_manager.cc:4895] TS e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245): delete failed for tablet b56b8dbd4e5b4892aa3b50409fc4584c because the tablet was not found. No further retry: Not found: Tablet not found: b56b8dbd4e5b4892aa3b50409fc4584c
I20250901 01:53:21.523027 755 catalog_manager.cc:6127] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5] alter complete (version 1)
I20250901 01:53:21.526017 1718 consensus_queue.cc:1035] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:21.528373 689 catalog_manager.cc:2149] created HMS catalog entry for table [id=49527ae170f04658b8b1eb01b07ffd04]
W20250901 01:53:21.529212 689 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 2c35f5a36eeb4a93aecdd71e2d1c3a21 is not leader of this config: current role FOLLOWER
I20250901 01:53:21.542948 755 catalog_manager.cc:6127] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: default.table_1ff7214e853748e58232b9bea0cb6912 [id=af5c975082de48cfa7295b2d8bd16635] alter complete (version 1)
I20250901 01:53:21.553437 755 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55080:
table { table_name: "default.Table_5d5e7ea4ac7d4e5f9de3237058d11e80" } modify_external_catalogs: true
I20250901 01:53:21.553721 755 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55080:
table { table_name: "default.Table_5d5e7ea4ac7d4e5f9de3237058d11e80" } modify_external_catalogs: true
I20250901 01:53:21.563306 1561 consensus_queue.cc:1035] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:21.608852 751 catalog_manager.cc:3460] renamed table default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98] in HMS: new name default.table_757f6c2777014fe1b0686fc1405b4b5f
I20250901 01:53:21.619426 682 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]
I20250901 01:53:21.620152 682 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:58182 (request call id 49) took 3192 ms. Trace:
I20250901 01:53:21.620304 682 rpcz_store.cc:276] 0901 01:53:18.427624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:19.554552 (+1126928us) service_pool.cc:225] Handling call
0901 01:53:21.018738 (+1464186us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:53:21.620048 (+601310us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":497531,"Hive Metastore.run_cpu_time_us":285,"Hive Metastore.run_wall_time_us":91008}
I20250901 01:53:21.678335 758 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19]
I20250901 01:53:21.681481 684 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=181eeac3e2394bcbaf3071e3fad223d8]
I20250901 01:53:21.681946 684 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58182 (request call id 52) took 3163 ms. Trace:
I20250901 01:53:21.682103 684 rpcz_store.cc:276] 0901 01:53:18.518761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:20.387797 (+1869036us) service_pool.cc:225] Handling call
0901 01:53:21.018021 (+630224us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:53:21.018843 (+ 822us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:53:21.412964 (+394121us) catalog_manager.cc:2148] Created new table in HMS catalog
0901 01:53:21.413794 (+ 830us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:21.413800 (+ 6us) write_op.cc:454] Released schema lock
0901 01:53:21.414065 (+ 265us) catalog_manager.cc:2156] Rolling back HMS table creation
0901 01:53:21.681890 (+267825us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:53:21.413690 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 2c35f5a36eeb4a93aecdd71e2d1c3a21 is not leader of this config: current role FOLLOWER)
0901 01:53:21.413707 (+ 17us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:21.413717 (+ 10us) write_op.cc:454] Released schema lock
0901 01:53:21.413724 (+ 7us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":415609,"Hive Metastore.run_cpu_time_us":727,"Hive Metastore.run_wall_time_us":245040,"child_traces":[["op",{}]]}
I20250901 01:53:21.727022 689 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=49527ae170f04658b8b1eb01b07ffd04]
I20250901 01:53:21.727519 689 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58182 (request call id 50) took 3298 ms. Trace:
I20250901 01:53:21.727658 689 rpcz_store.cc:276] 0901 01:53:18.428728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:19.645253 (+1216525us) service_pool.cc:225] Handling call
0901 01:53:21.019725 (+1374472us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:53:21.020312 (+ 587us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:53:21.528325 (+508013us) catalog_manager.cc:2148] Created new table in HMS catalog
0901 01:53:21.529048 (+ 723us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:21.529052 (+ 4us) write_op.cc:454] Released schema lock
0901 01:53:21.529326 (+ 274us) catalog_manager.cc:2156] Rolling back HMS table creation
0901 01:53:21.727424 (+198098us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:53:21.528971 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 2c35f5a36eeb4a93aecdd71e2d1c3a21 is not leader of this config: current role FOLLOWER)
0901 01:53:21.528984 (+ 13us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:21.528992 (+ 8us) write_op.cc:454] Released schema lock
0901 01:53:21.528996 (+ 4us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":544134,"Hive Metastore.run_cpu_time_us":602,"Hive Metastore.run_wall_time_us":161114,"child_traces":[["op",{}]]}
I20250901 01:53:21.728097 753 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647]
I20250901 01:53:21.747776 754 catalog_manager.cc:2149] created HMS catalog entry for table [id=d0ce834bce8e4a3fa795fb9e3b6b06a7]
I20250901 01:53:21.765465 754 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55080:
table { table_name: "default.Table_c95fbf8fa7a64aa18002f1e268cdea68" } new_table_name: "default.Table_9afa854fa1c148929b0ede8977435270" modify_external_catalogs: true
I20250901 01:53:21.776024 1147 tablet_service.cc:1468] Processing CreateTablet for tablet 3733efc4223d49a28f69cb598bdf7c94 (DEFAULT_TABLE table=default.table_facefcd0049d44a2b465d8315527aa08 [id=d0ce834bce8e4a3fa795fb9e3b6b06a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:21.776728 1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3733efc4223d49a28f69cb598bdf7c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:21.777401 1287 tablet_service.cc:1468] Processing CreateTablet for tablet 3733efc4223d49a28f69cb598bdf7c94 (DEFAULT_TABLE table=default.table_facefcd0049d44a2b465d8315527aa08 [id=d0ce834bce8e4a3fa795fb9e3b6b06a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:21.777927 1287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3733efc4223d49a28f69cb598bdf7c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:21.779122 1002 tablet_service.cc:1468] Processing CreateTablet for tablet 3733efc4223d49a28f69cb598bdf7c94 (DEFAULT_TABLE table=default.table_facefcd0049d44a2b465d8315527aa08 [id=d0ce834bce8e4a3fa795fb9e3b6b06a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:21.779616 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3733efc4223d49a28f69cb598bdf7c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:21.783986 756 catalog_manager.cc:2149] created HMS catalog entry for table [id=37fc4ed840564a8780eeb9847b373712]
I20250901 01:53:21.786412 1724 tablet_bootstrap.cc:492] T 3733efc4223d49a28f69cb598bdf7c94 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:21.788321 1724 tablet_bootstrap.cc:654] T 3733efc4223d49a28f69cb598bdf7c94 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:21.797806 1724 tablet_bootstrap.cc:492] T 3733efc4223d49a28f69cb598bdf7c94 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:21.798188 1724 ts_tablet_manager.cc:1397] T 3733efc4223d49a28f69cb598bdf7c94 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20250901 01:53:21.802560 756 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55080:
table { table_name: "default.Table_1ff7214e853748e58232b9bea0cb6912" } new_table_name: "default.Table_1b2b89cc1bc2459b80fd9de967ebd9a1" modify_external_catalogs: true
I20250901 01:53:21.804723 1724 raft_consensus.cc:357] T 3733efc4223d49a28f69cb598bdf7c94 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:21.805158 1724 raft_consensus.cc:383] T 3733efc4223d49a28f69cb598bdf7c94 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:21.805706 1724 raft_consensus.cc:738] T 3733efc4223d49a28f69cb598bdf7c94 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:21.806172 1724 consensus_queue.cc:260] T 3733efc4223d49a28f69cb598bdf7c94 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:21.808699 1724 ts_tablet_manager.cc:1428] T 3733efc4223d49a28f69cb598bdf7c94 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250901 01:53:21.822301 1722 tablet_bootstrap.cc:492] T 3733efc4223d49a28f69cb598bdf7c94 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:21.824010 1004 tablet_service.cc:1468] Processing CreateTablet for tablet b302bfaed5104909a1d18f7529398348 (DEFAULT_TABLE table=default.table_8082166db25745e78737b640063b3219 [id=37fc4ed840564a8780eeb9847b373712]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:21.824293 1722 tablet_bootstrap.cc:654] T 3733efc4223d49a28f69cb598bdf7c94 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:21.824286 1287 tablet_service.cc:1468] Processing CreateTablet for tablet b302bfaed5104909a1d18f7529398348 (DEFAULT_TABLE table=default.table_8082166db25745e78737b640063b3219 [id=37fc4ed840564a8780eeb9847b373712]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:21.824524 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b302bfaed5104909a1d18f7529398348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:21.824687 1287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b302bfaed5104909a1d18f7529398348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:21.831377 1146 tablet_service.cc:1468] Processing CreateTablet for tablet b302bfaed5104909a1d18f7529398348 (DEFAULT_TABLE table=default.table_8082166db25745e78737b640063b3219 [id=37fc4ed840564a8780eeb9847b373712]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:21.831876 1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b302bfaed5104909a1d18f7529398348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:21.836467 1713 tablet_bootstrap.cc:492] T b302bfaed5104909a1d18f7529398348 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:21.838133 1713 tablet_bootstrap.cc:654] T b302bfaed5104909a1d18f7529398348 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:21.838201 1722 tablet_bootstrap.cc:492] T 3733efc4223d49a28f69cb598bdf7c94 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:21.838415 1722 ts_tablet_manager.cc:1397] T 3733efc4223d49a28f69cb598bdf7c94 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.003s
I20250901 01:53:21.839157 750 catalog_manager.cc:2149] created HMS catalog entry for table [id=267f90dc52214e7986546d0e99a7f23a]
I20250901 01:53:21.847806 1722 raft_consensus.cc:357] T 3733efc4223d49a28f69cb598bdf7c94 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:21.848119 1722 raft_consensus.cc:383] T 3733efc4223d49a28f69cb598bdf7c94 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:21.848248 1722 raft_consensus.cc:738] T 3733efc4223d49a28f69cb598bdf7c94 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:21.848562 1722 consensus_queue.cc:260] T 3733efc4223d49a28f69cb598bdf7c94 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:21.849216 1722 ts_tablet_manager.cc:1428] T 3733efc4223d49a28f69cb598bdf7c94 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20250901 01:53:21.856060 1722 tablet_bootstrap.cc:492] T b302bfaed5104909a1d18f7529398348 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:21.856091 1713 tablet_bootstrap.cc:492] T b302bfaed5104909a1d18f7529398348 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:21.856305 1713 ts_tablet_manager.cc:1397] T b302bfaed5104909a1d18f7529398348 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.003s
I20250901 01:53:21.857214 1713 raft_consensus.cc:357] T b302bfaed5104909a1d18f7529398348 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:21.857460 1713 raft_consensus.cc:383] T b302bfaed5104909a1d18f7529398348 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:21.857551 1713 raft_consensus.cc:738] T b302bfaed5104909a1d18f7529398348 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:21.857810 1713 consensus_queue.cc:260] T b302bfaed5104909a1d18f7529398348 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:21.858748 1722 tablet_bootstrap.cc:654] T b302bfaed5104909a1d18f7529398348 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:21.859404 1713 ts_tablet_manager.cc:1428] T b302bfaed5104909a1d18f7529398348 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250901 01:53:21.860011 1713 tablet_bootstrap.cc:492] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:21.862286 1713 tablet_bootstrap.cc:654] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:21.864578 1724 tablet_bootstrap.cc:492] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:21.867179 1724 tablet_bootstrap.cc:654] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:21.873042 1724 tablet_bootstrap.cc:492] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:21.874990 1724 ts_tablet_manager.cc:1397] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20250901 01:53:21.875389 1686 raft_consensus.cc:491] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:21.875566 1686 raft_consensus.cc:513] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:21.876260 1686 leader_election.cc:290] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:21.876972 1166 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f81df1b890684c9cb01597c5bfebcefa" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:21.877292 1166 raft_consensus.cc:2466] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:21.877784 1305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f81df1b890684c9cb01597c5bfebcefa" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:21.878109 1305 raft_consensus.cc:2466] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:21.878129 953 leader_election.cc:304] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:21.878473 1724 raft_consensus.cc:357] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:21.878582 1686 raft_consensus.cc:2802] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:21.878681 1724 raft_consensus.cc:383] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:21.878695 1686 raft_consensus.cc:491] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:21.878783 1724 raft_consensus.cc:738] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:21.878791 1686 raft_consensus.cc:3058] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:21.879050 1724 consensus_queue.cc:260] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:21.882366 1686 raft_consensus.cc:513] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:21.883601 1166 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f81df1b890684c9cb01597c5bfebcefa" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:21.883842 1166 raft_consensus.cc:3058] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:21.893464 750 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55080:
table { table_name: "default.Table_7e006da3e6d24d309af545afeac11ca4" } new_table_name: "default.Table_746437fc94f24e50977140cddc823eaf" modify_external_catalogs: true
I20250901 01:53:21.894234 1002 tablet_service.cc:1468] Processing CreateTablet for tablet e4df70f8191e4115991d42178cb67bd7 (DEFAULT_TABLE table=default.table_bb66c7b604104254a48d435595b83fc4 [id=267f90dc52214e7986546d0e99a7f23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:21.894752 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4df70f8191e4115991d42178cb67bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:21.898603 1147 tablet_service.cc:1468] Processing CreateTablet for tablet e4df70f8191e4115991d42178cb67bd7 (DEFAULT_TABLE table=default.table_bb66c7b604104254a48d435595b83fc4 [id=267f90dc52214e7986546d0e99a7f23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:21.899084 1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4df70f8191e4115991d42178cb67bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:21.905902 1166 raft_consensus.cc:2466] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:21.906690 1287 tablet_service.cc:1468] Processing CreateTablet for tablet e4df70f8191e4115991d42178cb67bd7 (DEFAULT_TABLE table=default.table_bb66c7b604104254a48d435595b83fc4 [id=267f90dc52214e7986546d0e99a7f23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:21.907227 1287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4df70f8191e4115991d42178cb67bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:21.909806 953 leader_election.cc:304] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:21.910471 1686 leader_election.cc:290] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Requested vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:21.917599 1745 raft_consensus.cc:2802] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:21.920611 1745 raft_consensus.cc:695] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Becoming Leader. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Running, Role: LEADER
I20250901 01:53:21.921516 1745 consensus_queue.cc:237] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:21.922753 1713 tablet_bootstrap.cc:492] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:21.922969 1713 ts_tablet_manager.cc:1397] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.063s user 0.017s sys 0.016s
I20250901 01:53:21.923965 1713 raft_consensus.cc:357] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:21.924216 1713 raft_consensus.cc:383] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:21.924357 1713 raft_consensus.cc:738] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:21.924664 1724 ts_tablet_manager.cc:1428] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20250901 01:53:21.924650 1713 consensus_queue.cc:260] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:21.925275 1713 ts_tablet_manager.cc:1428] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 01:53:21.925664 1713 tablet_bootstrap.cc:492] T e4df70f8191e4115991d42178cb67bd7 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:21.927820 757 catalog_manager.cc:2149] created HMS catalog entry for table [id=5575a859d470414e9a1bf99d5738fc3a]
I20250901 01:53:21.931412 1305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f81df1b890684c9cb01597c5bfebcefa" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:21.931689 1305 raft_consensus.cc:3058] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:21.932010 1713 tablet_bootstrap.cc:654] T e4df70f8191e4115991d42178cb67bd7 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:21.932837 1722 tablet_bootstrap.cc:492] T b302bfaed5104909a1d18f7529398348 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:21.933035 1722 ts_tablet_manager.cc:1397] T b302bfaed5104909a1d18f7529398348 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.001s
I20250901 01:53:21.933975 1722 raft_consensus.cc:357] T b302bfaed5104909a1d18f7529398348 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:21.934399 1722 raft_consensus.cc:383] T b302bfaed5104909a1d18f7529398348 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:21.934509 1722 raft_consensus.cc:738] T b302bfaed5104909a1d18f7529398348 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:21.934789 1722 consensus_queue.cc:260] T b302bfaed5104909a1d18f7529398348 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:21.935500 1305 raft_consensus.cc:2466] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:21.939749 1722 ts_tablet_manager.cc:1428] T b302bfaed5104909a1d18f7529398348 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250901 01:53:21.942098 1722 tablet_bootstrap.cc:492] T e4df70f8191e4115991d42178cb67bd7 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:21.947988 757 catalog_manager.cc:5582] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 reported cstate change: term changed from 0 to 1, leader changed from <none> to b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b65b3154aefd49f7a37f70e9207e9853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } health_report { overall_health: HEALTHY } } }
I20250901 01:53:21.955391 1713 tablet_bootstrap.cc:492] T e4df70f8191e4115991d42178cb67bd7 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:21.955664 1713 ts_tablet_manager.cc:1397] T e4df70f8191e4115991d42178cb67bd7 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.002s
I20250901 01:53:21.956852 1713 raft_consensus.cc:357] T e4df70f8191e4115991d42178cb67bd7 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:21.957078 1713 raft_consensus.cc:383] T e4df70f8191e4115991d42178cb67bd7 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:21.957175 1713 raft_consensus.cc:738] T e4df70f8191e4115991d42178cb67bd7 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:21.957468 1713 consensus_queue.cc:260] T e4df70f8191e4115991d42178cb67bd7 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:21.958123 1713 ts_tablet_manager.cc:1428] T e4df70f8191e4115991d42178cb67bd7 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:53:21.972249 1722 tablet_bootstrap.cc:654] T e4df70f8191e4115991d42178cb67bd7 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:21.982328 1724 tablet_bootstrap.cc:492] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:21.984269 1724 tablet_bootstrap.cc:654] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:21.994652 759 catalog_manager.cc:2149] created HMS catalog entry for table [id=f868207bffb24b6795d0f636974d231d]
I20250901 01:53:21.997061 1147 tablet_service.cc:1468] Processing CreateTablet for tablet ad4ea5cf599f4719960fdf0b2e6fc054 (DEFAULT_TABLE table=default.table_9f3451ed25714464ad16b3e94214d0e3 [id=5575a859d470414e9a1bf99d5738fc3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:21.997290 1002 tablet_service.cc:1468] Processing CreateTablet for tablet ad4ea5cf599f4719960fdf0b2e6fc054 (DEFAULT_TABLE table=default.table_9f3451ed25714464ad16b3e94214d0e3 [id=5575a859d470414e9a1bf99d5738fc3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:21.997797 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad4ea5cf599f4719960fdf0b2e6fc054. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:21.997805 1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad4ea5cf599f4719960fdf0b2e6fc054. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.002625 1287 tablet_service.cc:1468] Processing CreateTablet for tablet ad4ea5cf599f4719960fdf0b2e6fc054 (DEFAULT_TABLE table=default.table_9f3451ed25714464ad16b3e94214d0e3 [id=5575a859d470414e9a1bf99d5738fc3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.002908 1686 raft_consensus.cc:491] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:22.003098 1686 raft_consensus.cc:513] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.003239 1287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad4ea5cf599f4719960fdf0b2e6fc054. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.003839 1686 leader_election.cc:290] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:22.004607 1305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3733efc4223d49a28f69cb598bdf7c94" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:22.004905 1305 raft_consensus.cc:2466] T 3733efc4223d49a28f69cb598bdf7c94 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:22.005246 1166 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3733efc4223d49a28f69cb598bdf7c94" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:22.005457 1166 raft_consensus.cc:2466] T 3733efc4223d49a28f69cb598bdf7c94 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:22.005585 954 leader_election.cc:304] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters:
I20250901 01:53:22.005960 1686 raft_consensus.cc:2802] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:22.006076 1686 raft_consensus.cc:491] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:22.006363 1686 raft_consensus.cc:3058] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.007117 1722 tablet_bootstrap.cc:492] T e4df70f8191e4115991d42178cb67bd7 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:22.007308 1722 ts_tablet_manager.cc:1397] T e4df70f8191e4115991d42178cb67bd7 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.000s
I20250901 01:53:22.008204 1722 raft_consensus.cc:357] T e4df70f8191e4115991d42178cb67bd7 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.008411 1722 raft_consensus.cc:383] T e4df70f8191e4115991d42178cb67bd7 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.008491 1722 raft_consensus.cc:738] T e4df70f8191e4115991d42178cb67bd7 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.008797 1722 consensus_queue.cc:260] T e4df70f8191e4115991d42178cb67bd7 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.009548 1722 ts_tablet_manager.cc:1428] T e4df70f8191e4115991d42178cb67bd7 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:53:22.025708 1686 raft_consensus.cc:513] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.026927 1724 tablet_bootstrap.cc:492] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:22.027163 1724 ts_tablet_manager.cc:1397] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.002s
I20250901 01:53:22.028019 1724 raft_consensus.cc:357] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.028250 1724 raft_consensus.cc:383] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.028321 1724 raft_consensus.cc:738] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.028472 1166 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3733efc4223d49a28f69cb598bdf7c94" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:22.028614 1724 consensus_queue.cc:260] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.028707 1166 raft_consensus.cc:3058] T 3733efc4223d49a28f69cb598bdf7c94 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.032931 1724 ts_tablet_manager.cc:1428] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250901 01:53:22.033181 1166 raft_consensus.cc:2466] T 3733efc4223d49a28f69cb598bdf7c94 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:22.034335 953 leader_election.cc:304] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:22.034742 1746 raft_consensus.cc:2802] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:22.034938 1746 raft_consensus.cc:695] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Becoming Leader. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Running, Role: LEADER
I20250901 01:53:22.035267 1746 consensus_queue.cc:237] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.036284 1713 tablet_bootstrap.cc:492] T ad4ea5cf599f4719960fdf0b2e6fc054 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:22.036238 1305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3733efc4223d49a28f69cb598bdf7c94" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:22.036437 1305 raft_consensus.cc:3058] T 3733efc4223d49a28f69cb598bdf7c94 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.038875 1713 tablet_bootstrap.cc:654] T ad4ea5cf599f4719960fdf0b2e6fc054 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.039776 1305 raft_consensus.cc:2466] T 3733efc4223d49a28f69cb598bdf7c94 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:22.044658 1686 leader_election.cc:290] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Requested vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:22.048918 1286 tablet_service.cc:1468] Processing CreateTablet for tablet 089fe953bba0403db288afefaf750c10 (DEFAULT_TABLE table=default.table_99b272393a1b40b4ae2d3bbaedc9ea60 [id=f868207bffb24b6795d0f636974d231d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.049687 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 089fe953bba0403db288afefaf750c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.049868 757 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55080:
table { table_name: "default.Table_da487bf0d57f439bb9da2ec57880a73b" } new_table_name: "default.Table_215810bf4c304b3cbce0d26ae7cfb7a0" modify_external_catalogs: true
I20250901 01:53:22.056586 1724 tablet_bootstrap.cc:492] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:22.057693 1146 tablet_service.cc:1468] Processing CreateTablet for tablet 089fe953bba0403db288afefaf750c10 (DEFAULT_TABLE table=default.table_99b272393a1b40b4ae2d3bbaedc9ea60 [id=f868207bffb24b6795d0f636974d231d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.058239 1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 089fe953bba0403db288afefaf750c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.066771 1724 tablet_bootstrap.cc:654] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.072984 1722 tablet_bootstrap.cc:492] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:22.075042 1722 tablet_bootstrap.cc:654] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.070514 1002 tablet_service.cc:1468] Processing CreateTablet for tablet 089fe953bba0403db288afefaf750c10 (DEFAULT_TABLE table=default.table_99b272393a1b40b4ae2d3bbaedc9ea60 [id=f868207bffb24b6795d0f636974d231d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.076581 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 089fe953bba0403db288afefaf750c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.072450 1713 tablet_bootstrap.cc:492] T ad4ea5cf599f4719960fdf0b2e6fc054 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:22.079360 1713 ts_tablet_manager.cc:1397] T ad4ea5cf599f4719960fdf0b2e6fc054 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.002s
I20250901 01:53:22.080374 1713 raft_consensus.cc:357] T ad4ea5cf599f4719960fdf0b2e6fc054 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.080693 1713 raft_consensus.cc:383] T ad4ea5cf599f4719960fdf0b2e6fc054 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.080833 1713 raft_consensus.cc:738] T ad4ea5cf599f4719960fdf0b2e6fc054 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.081193 1713 consensus_queue.cc:260] T ad4ea5cf599f4719960fdf0b2e6fc054 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.081861 1713 ts_tablet_manager.cc:1428] T ad4ea5cf599f4719960fdf0b2e6fc054 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:53:22.081686 759 catalog_manager.cc:5582] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 reported cstate change: term changed from 0 to 1, leader changed from <none> to b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b65b3154aefd49f7a37f70e9207e9853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } health_report { overall_health: UNKNOWN } } }
I20250901 01:53:22.085270 1724 tablet_bootstrap.cc:492] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:22.085470 1724 ts_tablet_manager.cc:1397] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20250901 01:53:22.086424 1724 raft_consensus.cc:357] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.086621 1724 raft_consensus.cc:383] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.086700 1724 raft_consensus.cc:738] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.091107 1724 consensus_queue.cc:260] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.096441 1724 ts_tablet_manager.cc:1428] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250901 01:53:22.096953 1724 tablet_bootstrap.cc:492] T 089fe953bba0403db288afefaf750c10 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:22.099090 1724 tablet_bootstrap.cc:654] T 089fe953bba0403db288afefaf750c10 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.103863 1722 tablet_bootstrap.cc:492] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:22.104082 1722 ts_tablet_manager.cc:1397] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250901 01:53:22.104985 1722 raft_consensus.cc:357] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.105221 1722 raft_consensus.cc:383] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.105319 1722 raft_consensus.cc:738] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.105551 1722 consensus_queue.cc:260] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.109949 1722 ts_tablet_manager.cc:1428] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250901 01:53:22.118649 1722 tablet_bootstrap.cc:492] T ad4ea5cf599f4719960fdf0b2e6fc054 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:22.120787 1722 tablet_bootstrap.cc:654] T ad4ea5cf599f4719960fdf0b2e6fc054 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.129470 1713 tablet_bootstrap.cc:492] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:22.131580 759 master_service.cc:948] ReplaceTablet: received request to replace tablet ca68c9a0c8fb4cccb081b404daff8b2a from {username='slave'} at 127.0.0.1:55080
I20250901 01:53:22.131940 1713 tablet_bootstrap.cc:654] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.132182 1724 tablet_bootstrap.cc:492] T 089fe953bba0403db288afefaf750c10 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:22.132416 1724 ts_tablet_manager.cc:1397] T 089fe953bba0403db288afefaf750c10 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.000s
I20250901 01:53:22.133440 1724 raft_consensus.cc:357] T 089fe953bba0403db288afefaf750c10 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.133697 1724 raft_consensus.cc:383] T 089fe953bba0403db288afefaf750c10 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.133778 1724 raft_consensus.cc:738] T 089fe953bba0403db288afefaf750c10 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.134182 1724 consensus_queue.cc:260] T 089fe953bba0403db288afefaf750c10 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.135049 1724 ts_tablet_manager.cc:1428] T 089fe953bba0403db288afefaf750c10 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250901 01:53:22.146314 759 catalog_manager.cc:5869] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Sending DeleteTablet for 3 replicas of tablet ca68c9a0c8fb4cccb081b404daff8b2a
I20250901 01:53:22.147333 1002 tablet_service.cc:1515] Processing DeleteTablet for tablet ca68c9a0c8fb4cccb081b404daff8b2a with delete_type TABLET_DATA_DELETED (replaced by tablet 3f1db382c27e455cae9ddf28a79ada2b) from {username='slave'} at 127.0.0.1:41698
I20250901 01:53:22.148501 1722 tablet_bootstrap.cc:492] T ad4ea5cf599f4719960fdf0b2e6fc054 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:22.148571 1147 tablet_service.cc:1515] Processing DeleteTablet for tablet ca68c9a0c8fb4cccb081b404daff8b2a with delete_type TABLET_DATA_DELETED (replaced by tablet 3f1db382c27e455cae9ddf28a79ada2b) from {username='slave'} at 127.0.0.1:57118
I20250901 01:53:22.148813 1722 ts_tablet_manager.cc:1397] T ad4ea5cf599f4719960fdf0b2e6fc054 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.004s
I20250901 01:53:22.149950 1722 raft_consensus.cc:357] T ad4ea5cf599f4719960fdf0b2e6fc054 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.151499 1287 tablet_service.cc:1515] Processing DeleteTablet for tablet ca68c9a0c8fb4cccb081b404daff8b2a with delete_type TABLET_DATA_DELETED (replaced by tablet 3f1db382c27e455cae9ddf28a79ada2b) from {username='slave'} at 127.0.0.1:53910
I20250901 01:53:22.157763 1713 tablet_bootstrap.cc:492] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:22.158318 1713 ts_tablet_manager.cc:1397] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.000s
I20250901 01:53:22.159626 1713 raft_consensus.cc:357] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.159955 1713 raft_consensus.cc:383] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.160054 1713 raft_consensus.cc:738] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.160422 1713 consensus_queue.cc:260] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.161089 1713 ts_tablet_manager.cc:1428] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250901 01:53:22.163702 1722 raft_consensus.cc:383] T ad4ea5cf599f4719960fdf0b2e6fc054 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.163902 1722 raft_consensus.cc:738] T ad4ea5cf599f4719960fdf0b2e6fc054 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.164340 1722 consensus_queue.cc:260] T ad4ea5cf599f4719960fdf0b2e6fc054 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.165076 1722 ts_tablet_manager.cc:1428] T ad4ea5cf599f4719960fdf0b2e6fc054 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.016s user 0.000s sys 0.003s
I20250901 01:53:22.176065 759 catalog_manager.cc:6652] ReplaceTablet: tablet ca68c9a0c8fb4cccb081b404daff8b2a deleted and replaced by tablet 3f1db382c27e455cae9ddf28a79ada2b
I20250901 01:53:22.188040 1004 tablet_service.cc:1468] Processing CreateTablet for tablet 3f1db382c27e455cae9ddf28a79ada2b (DEFAULT_TABLE table=default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.188233 1146 tablet_service.cc:1468] Processing CreateTablet for tablet 3f1db382c27e455cae9ddf28a79ada2b (DEFAULT_TABLE table=default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.188571 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1db382c27e455cae9ddf28a79ada2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.188884 1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1db382c27e455cae9ddf28a79ada2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.190644 1286 tablet_service.cc:1468] Processing CreateTablet for tablet 3f1db382c27e455cae9ddf28a79ada2b (DEFAULT_TABLE table=default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.191174 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1db382c27e455cae9ddf28a79ada2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.197851 1713 tablet_bootstrap.cc:492] T 3f1db382c27e455cae9ddf28a79ada2b P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:22.197994 752 catalog_manager.cc:2149] created HMS catalog entry for table [id=7a69995eebe446629422a7fe0924e58a]
I20250901 01:53:22.199857 1713 tablet_bootstrap.cc:654] T 3f1db382c27e455cae9ddf28a79ada2b P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
F20250901 01:53:22.211458 921 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 5 out of 13 received events: NotificationEvent(eventId=23, eventTime=1756691601, eventType=CREATE_TABLE, dbName=default, tableName=table_98a9d1400149426bab73e31aea263f5b, message={"server":"","servicePrincipal":"","db":"default","table":"table_98a9d1400149426bab73e31aea263f5b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_98a9d1400149426bab73e31aea263f5b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691601},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_98a9d1400149426bab73e31aea263f5b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_98a9d1400149426bab73e31aea263f5b\",\"kudu.table_id\":\"49527ae170f04658b8b1eb01b07ffd04\",\"transient_lastDdlTime\":\"1756691601\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1756691601, eventType=ALTER_TABLE, dbName=default, tableName=table_757f6c2777014fe1b0686fc1405b4b5f, message={"server":"","servicePrincipal":"","db":"default","table":"table_61dfd07a8f784854b5d9e104e010daaa","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_61dfd07a8f784854b5d9e104e010daaa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691599},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_61dfd07a8f784854b5d9e104e010daaa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_61dfd07a8f784854b5d9e104e010daaa\",\"kudu.table_id\":\"ae0da2fb2d1542308ccb9a95247c1d98\",\"transient_lastDdlTime\":\"1756691599\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_757f6c2777014fe1b0686fc1405b4b5f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691599},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_757f6c2777014fe1b0686fc1405b4b5f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_757f6c2777014fe1b0686fc1405b4b5f\",\"kudu.table_id\":\"ae0da2fb2d1542308ccb9a95247c1d98\",\"transient_lastDdlTime\":\"1756691599\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_c2864548fe1a4719b73cf55d0933e538, message={"server":"","servicePrincipal":"","db":"default","table":"table_c2864548fe1a4719b73cf55d0933e538","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c2864548fe1a4719b73cf55d0933e538\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691598},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_c2864548fe1a4719b73cf55d0933e538\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c2864548fe1a4719b73cf55d0933e538\",\"kudu.table_id\":\"31d721fb8b204568a0190dda6def01cd\",\"transient_lastDdlTime\":\"1756691598\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_56fde480a00a4e62b84eedba22b0bade, message={"server":"","servicePrincipal":"","db":"default","table":"table_56fde480a00a4e62b84eedba22b0bade","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_56fde480a00a4e62b84eedba22b0bade\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691601},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_56fde480a00a4e62b84eedba22b0bade\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_56fde480a00a4e62b84eedba22b0bade\",\"kudu.table_id\":\"181eeac3e2394bcbaf3071e3fad223d8\",\"transient_lastDdlTime\":\"1756691601\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_7722e49197be4ca9b9c7e67052f07c4b, message={"server":"","servicePrincipal":"","db":"default","table":"table_7722e49197be4ca9b9c7e67052f07c4b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_7722e49197be4ca9b9c7e67052f07c4b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691598},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_7722e49197be4ca9b9c7e67052f07c4b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7722e49197be4ca9b9c7e67052f07c4b\",\"kudu.table_id\":\"2e950743029d48a49f05207cd5faee19\",\"transient_lastDdlTime\":\"1756691598\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250901 01:53:22.215049 1722 tablet_bootstrap.cc:492] T 3f1db382c27e455cae9ddf28a79ada2b P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:22.216995 1724 tablet_bootstrap.cc:492] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:22.217228 1722 tablet_bootstrap.cc:654] T 3f1db382c27e455cae9ddf28a79ada2b P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.219357 1718 raft_consensus.cc:491] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:22.219650 1718 raft_consensus.cc:513] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.220379 1724 tablet_bootstrap.cc:654] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.220793 1718 leader_election.cc:290] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:22.221952 1166 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad4ea5cf599f4719960fdf0b2e6fc054" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:22.222337 1166 raft_consensus.cc:2466] T ad4ea5cf599f4719960fdf0b2e6fc054 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:22.223670 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad4ea5cf599f4719960fdf0b2e6fc054" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:22.224033 1022 raft_consensus.cc:2466] T ad4ea5cf599f4719960fdf0b2e6fc054 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:22.225015 1240 leader_election.cc:304] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:22.226181 1718 raft_consensus.cc:491] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:22.226431 1718 raft_consensus.cc:513] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.227352 1718 leader_election.cc:290] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:22.227567 1718 raft_consensus.cc:2802] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:22.227684 1718 raft_consensus.cc:491] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:22.227794 1718 raft_consensus.cc:3058] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.225260 752 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55080 (request call id 32) took 1161 ms. Trace:
I20250901 01:53:22.231047 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b302bfaed5104909a1d18f7529398348" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:22.229756 752 rpcz_store.cc:276] 0901 01:53:21.063694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:53:21.182996 (+119302us) service_pool.cc:225] Handling call
0901 01:53:21.437904 (+254908us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:53:21.438388 (+ 484us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:53:22.197927 (+759539us) catalog_manager.cc:2148] Created new table in HMS catalog
0901 01:53:22.224924 (+ 26997us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:53:22.225055 (+ 131us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:53:22.225205 (+ 150us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:53:22.198918 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:53:22.199050 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:53:22.199058 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:53:22.199061 (+ 3us) tablet.cc:660] Decoding operations
0901 01:53:22.199183 (+ 122us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:53:22.199202 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0901 01:53:22.199211 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:53:22.199326 (+ 115us) tablet.cc:699] Row locks acquired
0901 01:53:22.199329 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:53:22.199372 (+ 43us) write_op.cc:270] Start()
0901 01:53:22.199412 (+ 40us) write_op.cc:276] Timestamp: P: 1756691602199356 usec, L: 0
0901 01:53:22.199417 (+ 5us) op_driver.cc:348] REPLICATION: starting
0901 01:53:22.199644 (+ 227us) log.cc:844] Serialized 432 byte log entry
0901 01:53:22.223849 (+ 24205us) op_driver.cc:464] REPLICATION: finished
0901 01:53:22.224027 (+ 178us) write_op.cc:301] APPLY: starting
0901 01:53:22.224053 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
0901 01:53:22.224132 (+ 79us) tablet.cc:1368] finished BulkCheckPresence
0901 01:53:22.224186 (+ 54us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:53:22.224342 (+ 156us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:53:22.224400 (+ 58us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:53:22.224412 (+ 12us) write_op.cc:312] APPLY: finished
0901 01:53:22.224538 (+ 126us) log.cc:844] Serialized 28 byte log entry
0901 01:53:22.224625 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:22.224652 (+ 27us) write_op.cc:454] Released schema lock
0901 01:53:22.224660 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0901 01:53:22.224750 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:53:22.224755 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":555882,"Hive Metastore.run_cpu_time_us":359,"Hive Metastore.run_wall_time_us":203342,"child_traces":[["op",{"apply.queue_time_us":108,"apply.run_cpu_time_us":707,"apply.run_wall_time_us":706,"num_ops":2,"prepare.queue_time_us":119,"prepare.run_cpu_time_us":864,"prepare.run_wall_time_us":864,"raft.queue_time_us":79,"raft.run_cpu_time_us":463,"raft.run_wall_time_us":461,"replication_time_us":24376,"spinlock_wait_cycles":129024}]]}
I20250901 01:53:22.231376 1022 raft_consensus.cc:2466] T b302bfaed5104909a1d18f7529398348 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:22.232214 1243 leader_election.cc:304] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters:
@ 0x7f60c7cf6dcd google::LogMessage::Fail() at ??:0
I20250901 01:53:22.243814 759 master_service.cc:948] ReplaceTablet: received request to replace tablet 138d19e14b734c1cace0d7ba51ab2caa from {username='slave'} at 127.0.0.1:55080
@ 0x7f60c7cfab93 google::LogMessage::SendToLog() at ??:0
@ 0x7f60c7cf67cc google::LogMessage::Flush() at ??:0
I20250901 01:53:22.260171 759 catalog_manager.cc:5869] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Sending DeleteTablet for 3 replicas of tablet 138d19e14b734c1cace0d7ba51ab2caa
I20250901 01:53:22.261134 1004 tablet_service.cc:1515] Processing DeleteTablet for tablet 138d19e14b734c1cace0d7ba51ab2caa with delete_type TABLET_DATA_DELETED (replaced by tablet debd885f996642faa60cfa5af0283b5f) from {username='slave'} at 127.0.0.1:41698
I20250901 01:53:22.261307 1286 tablet_service.cc:1515] Processing DeleteTablet for tablet 138d19e14b734c1cace0d7ba51ab2caa with delete_type TABLET_DATA_DELETED (replaced by tablet debd885f996642faa60cfa5af0283b5f) from {username='slave'} at 127.0.0.1:53910
I20250901 01:53:22.261543 759 catalog_manager.cc:6652] ReplaceTablet: tablet 138d19e14b734c1cace0d7ba51ab2caa deleted and replaced by tablet debd885f996642faa60cfa5af0283b5f
I20250901 01:53:22.261787 759 master_service.cc:948] ReplaceTablet: received request to replace tablet 9cd7993f6a99468caa9b13fc4e8c0ba1 from {username='slave'} at 127.0.0.1:55080
I20250901 01:53:22.265128 1146 tablet_service.cc:1515] Processing DeleteTablet for tablet 138d19e14b734c1cace0d7ba51ab2caa with delete_type TABLET_DATA_DELETED (replaced by tablet debd885f996642faa60cfa5af0283b5f) from {username='slave'} at 127.0.0.1:57118
I20250901 01:53:22.269183 1765 raft_consensus.cc:491] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:22.269459 1765 raft_consensus.cc:513] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.270466 1765 leader_election.cc:290] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:22.271929 1765 raft_consensus.cc:2802] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:22.272040 1765 raft_consensus.cc:491] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:22.272133 1765 raft_consensus.cc:3058] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.273331 1004 tablet_service.cc:1468] Processing CreateTablet for tablet 6b2f0235fadf41ed8e705af5059c4702 (DEFAULT_TABLE table=default.table_167322850c734684b896b875526a5c6e [id=7a69995eebe446629422a7fe0924e58a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.273893 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b2f0235fadf41ed8e705af5059c4702. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:53:22.273164 1240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35245, user_credentials={real_user=slave}} blocked reactor thread for 48188.8us
I20250901 01:53:22.274900 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4df70f8191e4115991d42178cb67bd7" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:22.275182 1022 raft_consensus.cc:2466] T e4df70f8191e4115991d42178cb67bd7 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:22.275784 1243 leader_election.cc:304] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters:
I20250901 01:53:22.276561 1766 raft_consensus.cc:2802] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:22.276690 1766 raft_consensus.cc:491] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:22.276834 1766 raft_consensus.cc:3058] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.278393 1724 tablet_bootstrap.cc:492] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:22.278581 1724 ts_tablet_manager.cc:1397] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.003s
I20250901 01:53:22.279783 1724 raft_consensus.cc:357] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.280047 1724 raft_consensus.cc:383] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.280150 1724 raft_consensus.cc:738] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.280558 1724 consensus_queue.cc:260] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.281806 1724 ts_tablet_manager.cc:1428] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250901 01:53:22.282763 1166 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b302bfaed5104909a1d18f7529398348" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:22.283067 1166 raft_consensus.cc:2466] T b302bfaed5104909a1d18f7529398348 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:22.283075 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4df70f8191e4115991d42178cb67bd7" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:22.283547 1167 raft_consensus.cc:2466] T e4df70f8191e4115991d42178cb67bd7 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:22.284803 1718 raft_consensus.cc:513] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.285771 1718 leader_election.cc:290] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Requested vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:22.289153 1765 raft_consensus.cc:513] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.289523 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad4ea5cf599f4719960fdf0b2e6fc054" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:22.289839 1022 raft_consensus.cc:3058] T ad4ea5cf599f4719960fdf0b2e6fc054 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.296216 1765 leader_election.cc:290] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Requested vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:22.300433 752 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55080:
table { table_name: "default.Table_8f21a26e011c4ed1b1ae7cb364e09195" } modify_external_catalogs: true
I20250901 01:53:22.300886 752 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55080:
table { table_name: "default.Table_8f21a26e011c4ed1b1ae7cb364e09195" } modify_external_catalogs: true
I20250901 01:53:22.304574 1146 tablet_service.cc:1468] Processing CreateTablet for tablet 6b2f0235fadf41ed8e705af5059c4702 (DEFAULT_TABLE table=default.table_167322850c734684b896b875526a5c6e [id=7a69995eebe446629422a7fe0924e58a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.305142 1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b2f0235fadf41ed8e705af5059c4702. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.305261 1286 tablet_service.cc:1468] Processing CreateTablet for tablet 6b2f0235fadf41ed8e705af5059c4702 (DEFAULT_TABLE table=default.table_167322850c734684b896b875526a5c6e [id=7a69995eebe446629422a7fe0924e58a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.305691 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b2f0235fadf41ed8e705af5059c4702. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.315680 759 catalog_manager.cc:5869] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Sending DeleteTablet for 3 replicas of tablet 9cd7993f6a99468caa9b13fc4e8c0ba1
I20250901 01:53:22.316411 759 catalog_manager.cc:6652] ReplaceTablet: tablet 9cd7993f6a99468caa9b13fc4e8c0ba1 deleted and replaced by tablet de13eb65b0df4b8ebc049ff753123d53
I20250901 01:53:22.316525 1023 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b302bfaed5104909a1d18f7529398348" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:22.316843 1023 raft_consensus.cc:3058] T b302bfaed5104909a1d18f7529398348 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.316933 759 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55080:
name: "default.Table_820e14b2e7cf4bd7ad59ea83d8cb4d60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:22.318444 1285 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cd7993f6a99468caa9b13fc4e8c0ba1 with delete_type TABLET_DATA_DELETED (replaced by tablet de13eb65b0df4b8ebc049ff753123d53) from {username='slave'} at 127.0.0.1:53910
I20250901 01:53:22.319671 1003 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cd7993f6a99468caa9b13fc4e8c0ba1 with delete_type TABLET_DATA_DELETED (replaced by tablet de13eb65b0df4b8ebc049ff753123d53) from {username='slave'} at 127.0.0.1:41698
I20250901 01:53:22.320827 1146 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cd7993f6a99468caa9b13fc4e8c0ba1 with delete_type TABLET_DATA_DELETED (replaced by tablet de13eb65b0df4b8ebc049ff753123d53) from {username='slave'} at 127.0.0.1:57118
I20250901 01:53:22.322191 1724 tablet_bootstrap.cc:492] T 6b2f0235fadf41ed8e705af5059c4702 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
@ 0x7f60c7cf6a11 google::LogMessage::~LogMessage() at ??:0
I20250901 01:53:22.324363 1724 tablet_bootstrap.cc:654] T 6b2f0235fadf41ed8e705af5059c4702 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.327248 1713 tablet_bootstrap.cc:492] T 3f1db382c27e455cae9ddf28a79ada2b P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:22.327427 1713 ts_tablet_manager.cc:1397] T 3f1db382c27e455cae9ddf28a79ada2b P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.130s user 0.003s sys 0.000s
I20250901 01:53:22.328346 1713 raft_consensus.cc:357] T 3f1db382c27e455cae9ddf28a79ada2b P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.328568 1713 raft_consensus.cc:383] T 3f1db382c27e455cae9ddf28a79ada2b P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.328665 1713 raft_consensus.cc:738] T 3f1db382c27e455cae9ddf28a79ada2b P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.328922 1713 consensus_queue.cc:260] T 3f1db382c27e455cae9ddf28a79ada2b P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.329507 1713 ts_tablet_manager.cc:1428] T 3f1db382c27e455cae9ddf28a79ada2b P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250901 01:53:22.331399 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad4ea5cf599f4719960fdf0b2e6fc054" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:22.331665 1167 raft_consensus.cc:3058] T ad4ea5cf599f4719960fdf0b2e6fc054 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.333387 1713 tablet_bootstrap.cc:492] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:22.335856 1166 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b302bfaed5104909a1d18f7529398348" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:22.336086 1166 raft_consensus.cc:3058] T b302bfaed5104909a1d18f7529398348 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.336376 1713 tablet_bootstrap.cc:654] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.341372 1146 tablet_service.cc:1468] Processing CreateTablet for tablet debd885f996642faa60cfa5af0283b5f (DEFAULT_TABLE table=default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.341936 1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet debd885f996642faa60cfa5af0283b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.353348 1286 tablet_service.cc:1468] Processing CreateTablet for tablet debd885f996642faa60cfa5af0283b5f (DEFAULT_TABLE table=default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.355840 1003 tablet_service.cc:1468] Processing CreateTablet for tablet de13eb65b0df4b8ebc049ff753123d53 (DEFAULT_TABLE table=default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.356463 1003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de13eb65b0df4b8ebc049ff753123d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.361518 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet debd885f996642faa60cfa5af0283b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.362737 1004 tablet_service.cc:1468] Processing CreateTablet for tablet debd885f996642faa60cfa5af0283b5f (DEFAULT_TABLE table=default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.362958 1713 tablet_bootstrap.cc:492] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:22.363143 1713 ts_tablet_manager.cc:1397] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.005s
I20250901 01:53:22.363281 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet debd885f996642faa60cfa5af0283b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.364015 1713 raft_consensus.cc:357] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.364233 1713 raft_consensus.cc:383] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.364305 1713 raft_consensus.cc:738] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.367744 1145 tablet_service.cc:1468] Processing CreateTablet for tablet de13eb65b0df4b8ebc049ff753123d53 (DEFAULT_TABLE table=default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.368500 1722 tablet_bootstrap.cc:492] T 3f1db382c27e455cae9ddf28a79ada2b P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:22.368518 1145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de13eb65b0df4b8ebc049ff753123d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.368701 1722 ts_tablet_manager.cc:1397] T 3f1db382c27e455cae9ddf28a79ada2b P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.154s user 0.011s sys 0.001s
I20250901 01:53:22.369786 1713 consensus_queue.cc:260] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.369916 1722 raft_consensus.cc:357] T 3f1db382c27e455cae9ddf28a79ada2b P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.370178 1722 raft_consensus.cc:383] T 3f1db382c27e455cae9ddf28a79ada2b P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.372227 1285 tablet_service.cc:1468] Processing CreateTablet for tablet de13eb65b0df4b8ebc049ff753123d53 (DEFAULT_TABLE table=default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:22.372766 1285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de13eb65b0df4b8ebc049ff753123d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:22.376181 1724 tablet_bootstrap.cc:492] T 6b2f0235fadf41ed8e705af5059c4702 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:22.376363 1724 ts_tablet_manager.cc:1397] T 6b2f0235fadf41ed8e705af5059c4702 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.011s
I20250901 01:53:22.377270 1724 raft_consensus.cc:357] T 6b2f0235fadf41ed8e705af5059c4702 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.377494 1724 raft_consensus.cc:383] T 6b2f0235fadf41ed8e705af5059c4702 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.378135 1722 raft_consensus.cc:738] T 3f1db382c27e455cae9ddf28a79ada2b P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.378537 1713 ts_tablet_manager.cc:1428] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20250901 01:53:22.378502 1722 consensus_queue.cc:260] T 3f1db382c27e455cae9ddf28a79ada2b P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.384639 1722 ts_tablet_manager.cc:1428] T 3f1db382c27e455cae9ddf28a79ada2b P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.016s user 0.001s sys 0.004s
I20250901 01:53:22.385655 1724 raft_consensus.cc:738] T 6b2f0235fadf41ed8e705af5059c4702 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.386078 1724 consensus_queue.cc:260] T 6b2f0235fadf41ed8e705af5059c4702 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.386727 1724 ts_tablet_manager.cc:1428] T 6b2f0235fadf41ed8e705af5059c4702 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20250901 01:53:22.388866 1722 tablet_bootstrap.cc:492] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:22.389335 1022 raft_consensus.cc:2466] T ad4ea5cf599f4719960fdf0b2e6fc054 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:22.391168 1722 tablet_bootstrap.cc:654] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.394326 1167 raft_consensus.cc:2466] T ad4ea5cf599f4719960fdf0b2e6fc054 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:22.395349 1240 leader_election.cc:304] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:22.397706 1765 raft_consensus.cc:2802] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:22.398041 1765 raft_consensus.cc:695] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [term 1 LEADER]: Becoming Leader. State: Replica: ad2ba2450480454388723cd729cc438e, State: Running, Role: LEADER
I20250901 01:53:22.398479 1765 consensus_queue.cc:237] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.398999 1759 tablet_replica.cc:331] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853: stopping tablet replica
I20250901 01:53:22.399230 1759 raft_consensus.cc:2241] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:53:22.399619 1759 raft_consensus.cc:2270] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:22.400313 1759 ts_tablet_manager.cc:1905] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:22.403486 1760 tablet_replica.cc:331] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5: stopping tablet replica
I20250901 01:53:22.403771 1760 raft_consensus.cc:2241] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:22.404078 1760 raft_consensus.cc:2270] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:22.404881 1760 ts_tablet_manager.cc:1905] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:22.404789 1766 raft_consensus.cc:513] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.405679 1766 leader_election.cc:290] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Requested vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
@ 0x7f60de12b794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250901 01:53:22.406303 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4df70f8191e4115991d42178cb67bd7" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:22.407068 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4df70f8191e4115991d42178cb67bd7" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:22.407199 1167 raft_consensus.cc:3058] T e4df70f8191e4115991d42178cb67bd7 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.407312 1022 raft_consensus.cc:3058] T e4df70f8191e4115991d42178cb67bd7 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.410260 1761 tablet_replica.cc:331] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e: stopping tablet replica
I20250901 01:53:22.410511 1761 raft_consensus.cc:2241] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:22.410735 1761 raft_consensus.cc:2270] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:22.411347 1761 ts_tablet_manager.cc:1905] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:22.414969 1722 tablet_bootstrap.cc:492] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:22.415371 1722 ts_tablet_manager.cc:1397] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250901 01:53:22.416445 1722 raft_consensus.cc:357] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.416684 1722 raft_consensus.cc:383] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.416769 1722 raft_consensus.cc:738] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.417085 1722 consensus_queue.cc:260] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
@ 0x7f60de12a041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250901 01:53:22.420925 1722 ts_tablet_manager.cc:1428] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
@ 0x7f60de131cbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f60de131b39 std::_Function_handler<>::_M_invoke() at ??:0
I20250901 01:53:22.428576 1724 tablet_bootstrap.cc:492] T de13eb65b0df4b8ebc049ff753123d53 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:22.430510 1724 tablet_bootstrap.cc:654] T de13eb65b0df4b8ebc049ff753123d53 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
@ 0xa774dd std::function<>::operator()() at ??:?
I20250901 01:53:22.442988 1724 tablet_bootstrap.cc:492] T de13eb65b0df4b8ebc049ff753123d53 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:22.443153 1724 ts_tablet_manager.cc:1397] T de13eb65b0df4b8ebc049ff753123d53 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20250901 01:53:22.443936 1724 raft_consensus.cc:357] T de13eb65b0df4b8ebc049ff753123d53 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.444118 1724 raft_consensus.cc:383] T de13eb65b0df4b8ebc049ff753123d53 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.444192 1724 raft_consensus.cc:738] T de13eb65b0df4b8ebc049ff753123d53 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.444468 1724 consensus_queue.cc:260] T de13eb65b0df4b8ebc049ff753123d53 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.445068 1724 ts_tablet_manager.cc:1428] T de13eb65b0df4b8ebc049ff753123d53 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:53:22.448508 1724 tablet_bootstrap.cc:492] T debd885f996642faa60cfa5af0283b5f P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
@ 0x7f60caa53cc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f60cde756db start_thread at ??:0
@ 0x7f60c6ba771f clone at ??:0
I20250901 01:53:22.459131 1724 tablet_bootstrap.cc:654] T debd885f996642faa60cfa5af0283b5f P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.468989 1765 raft_consensus.cc:491] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:22.469223 1765 raft_consensus.cc:513] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.470194 1724 tablet_bootstrap.cc:492] T debd885f996642faa60cfa5af0283b5f P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:22.470423 1724 ts_tablet_manager.cc:1397] T debd885f996642faa60cfa5af0283b5f P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.005s
I20250901 01:53:22.471065 1024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f1db382c27e455cae9ddf28a79ada2b" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:22.471292 1024 raft_consensus.cc:2466] T 3f1db382c27e455cae9ddf28a79ada2b P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:22.471691 1165 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f1db382c27e455cae9ddf28a79ada2b" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:22.471915 1165 raft_consensus.cc:2466] T 3f1db382c27e455cae9ddf28a79ada2b P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 0.
I20250901 01:53:22.472254 1724 raft_consensus.cc:357] T debd885f996642faa60cfa5af0283b5f P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.472416 1240 leader_election.cc:304] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:22.472518 1724 raft_consensus.cc:383] T debd885f996642faa60cfa5af0283b5f P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.472604 1724 raft_consensus.cc:738] T debd885f996642faa60cfa5af0283b5f P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.472733 1766 raft_consensus.cc:2802] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:22.472831 1766 raft_consensus.cc:491] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:22.472903 1724 consensus_queue.cc:260] T debd885f996642faa60cfa5af0283b5f P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.472923 1766 raft_consensus.cc:3058] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.475936 1724 ts_tablet_manager.cc:1428] T debd885f996642faa60cfa5af0283b5f P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250901 01:53:22.470081 1765 leader_election.cc:290] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:22.480443 1023 raft_consensus.cc:2466] T b302bfaed5104909a1d18f7529398348 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:22.480918 1166 raft_consensus.cc:2466] T b302bfaed5104909a1d18f7529398348 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:22.481273 1243 leader_election.cc:304] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters:
I20250901 01:53:22.484226 1765 raft_consensus.cc:2802] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:22.484383 1765 raft_consensus.cc:695] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [term 1 LEADER]: Becoming Leader. State: Replica: ad2ba2450480454388723cd729cc438e, State: Running, Role: LEADER
I20250901 01:53:22.484655 1765 consensus_queue.cc:237] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.485996 1766 raft_consensus.cc:513] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.486616 1713 tablet_bootstrap.cc:492] T debd885f996642faa60cfa5af0283b5f P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:22.486783 1766 leader_election.cc:290] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Requested vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:22.489599 1713 tablet_bootstrap.cc:654] T debd885f996642faa60cfa5af0283b5f P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.490736 1023 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f1db382c27e455cae9ddf28a79ada2b" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:22.491020 1023 raft_consensus.cc:3058] T 3f1db382c27e455cae9ddf28a79ada2b P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.492635 1166 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f1db382c27e455cae9ddf28a79ada2b" candidate_uuid: "ad2ba2450480454388723cd729cc438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:22.492882 1166 raft_consensus.cc:3058] T 3f1db382c27e455cae9ddf28a79ada2b P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.497583 1713 tablet_bootstrap.cc:492] T debd885f996642faa60cfa5af0283b5f P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:22.497773 1713 ts_tablet_manager.cc:1397] T debd885f996642faa60cfa5af0283b5f P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20250901 01:53:22.498764 1713 raft_consensus.cc:357] T debd885f996642faa60cfa5af0283b5f P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.498965 1713 raft_consensus.cc:383] T debd885f996642faa60cfa5af0283b5f P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.499058 1713 raft_consensus.cc:738] T debd885f996642faa60cfa5af0283b5f P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.499403 1713 consensus_queue.cc:260] T debd885f996642faa60cfa5af0283b5f P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.503597 1713 ts_tablet_manager.cc:1428] T debd885f996642faa60cfa5af0283b5f P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250901 01:53:22.521667 1167 raft_consensus.cc:2466] T e4df70f8191e4115991d42178cb67bd7 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:22.521907 1761 ts_tablet_manager.cc:1918] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:22.522078 1761 log.cc:1199] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal/wals/ca68c9a0c8fb4cccb081b404daff8b2a
I20250901 01:53:22.522446 1759 ts_tablet_manager.cc:1918] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:22.522627 1759 log.cc:1199] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/wals/ca68c9a0c8fb4cccb081b404daff8b2a
I20250901 01:53:22.522644 1761 ts_tablet_manager.cc:1939] T ca68c9a0c8fb4cccb081b404daff8b2a P ad2ba2450480454388723cd729cc438e: Deleting consensus metadata
I20250901 01:53:22.523165 1759 ts_tablet_manager.cc:1939] T ca68c9a0c8fb4cccb081b404daff8b2a P b65b3154aefd49f7a37f70e9207e9853: Deleting consensus metadata
I20250901 01:53:22.523211 1760 ts_tablet_manager.cc:1918] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:22.523348 1760 log.cc:1199] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal/wals/ca68c9a0c8fb4cccb081b404daff8b2a
I20250901 01:53:22.523805 1760 ts_tablet_manager.cc:1939] T ca68c9a0c8fb4cccb081b404daff8b2a P e3c774a4edf44d6098a2332e734967a5: Deleting consensus metadata
I20250901 01:53:22.524060 1022 raft_consensus.cc:2466] T e4df70f8191e4115991d42178cb67bd7 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:22.525245 1722 tablet_bootstrap.cc:492] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:22.526903 1761 tablet_replica.cc:331] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e: stopping tablet replica
I20250901 01:53:22.527154 1761 raft_consensus.cc:2241] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:22.527396 1722 tablet_bootstrap.cc:654] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.527406 1761 raft_consensus.cc:2270] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:22.528076 1761 ts_tablet_manager.cc:1905] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:22.529088 1240 leader_election.cc:304] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:22.529482 1765 raft_consensus.cc:2802] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:22.529654 1765 raft_consensus.cc:695] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [term 1 LEADER]: Becoming Leader. State: Replica: ad2ba2450480454388723cd729cc438e, State: Running, Role: LEADER
I20250901 01:53:22.529966 1765 consensus_queue.cc:237] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.532227 1759 tablet_replica.cc:331] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853: stopping tablet replica
I20250901 01:53:22.532506 1759 raft_consensus.cc:2241] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:22.532712 1759 raft_consensus.cc:2270] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:22.533373 1759 ts_tablet_manager.cc:1905] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:22.537449 1713 tablet_bootstrap.cc:492] T de13eb65b0df4b8ebc049ff753123d53 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:22.539484 1713 tablet_bootstrap.cc:654] T de13eb65b0df4b8ebc049ff753123d53 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.540513 1760 tablet_replica.cc:331] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5: stopping tablet replica
I20250901 01:53:22.540781 1760 raft_consensus.cc:2241] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:53:22.541425 1760 raft_consensus.cc:2270] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:22.541476 1722 tablet_bootstrap.cc:492] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:22.541709 1722 ts_tablet_manager.cc:1397] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20250901 01:53:22.542641 1760 ts_tablet_manager.cc:1905] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:22.542934 1722 raft_consensus.cc:357] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.543237 1722 raft_consensus.cc:383] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.543370 1722 raft_consensus.cc:738] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.543745 1722 consensus_queue.cc:260] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.551376 1722 ts_tablet_manager.cc:1428] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250901 01:53:22.551815 1722 tablet_bootstrap.cc:492] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:22.553730 1722 tablet_bootstrap.cc:654] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:22.558794 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task1fHXBS/build/asan/bin/kudu pid 725
I20250901 01:53:22.560914 1669 raft_consensus.cc:491] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:22.561179 1669 raft_consensus.cc:513] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.563781 1669 leader_election.cc:290] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2ba2450480454388723cd729cc438e (127.0.104.3:42927), b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:22.565001 1166 raft_consensus.cc:2466] T 3f1db382c27e455cae9ddf28a79ada2b P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:22.565227 1722 tablet_bootstrap.cc:492] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:22.565426 1722 ts_tablet_manager.cc:1397] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20250901 01:53:22.565609 1240 leader_election.cc:304] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:22.566605 1722 raft_consensus.cc:357] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.566872 1722 raft_consensus.cc:383] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.566957 1722 raft_consensus.cc:738] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.567322 1765 raft_consensus.cc:2802] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:22.567512 1765 raft_consensus.cc:695] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [term 1 LEADER]: Becoming Leader. State: Replica: ad2ba2450480454388723cd729cc438e, State: Running, Role: LEADER
I20250901 01:53:22.567885 1765 consensus_queue.cc:237] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.569780 1722 consensus_queue.cc:260] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.570686 1305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "089fe953bba0403db288afefaf750c10" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:22.571398 1305 raft_consensus.cc:2466] T 089fe953bba0403db288afefaf750c10 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 0.
I20250901 01:53:22.572798 1722 ts_tablet_manager.cc:1428] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250901 01:53:22.574935 1101 leader_election.cc:304] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:22.575515 1669 raft_consensus.cc:2802] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:22.575661 1669 raft_consensus.cc:491] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:22.575788 1669 raft_consensus.cc:3058] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.583523 1686 raft_consensus.cc:491] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:22.583736 1686 raft_consensus.cc:513] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.584470 1686 leader_election.cc:290] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:22.586297 1305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "089fe953bba0403db288afefaf750c10" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:22.586575 1305 raft_consensus.cc:2466] T 089fe953bba0403db288afefaf750c10 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:22.586678 1166 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "089fe953bba0403db288afefaf750c10" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:22.587369 954 leader_election.cc:304] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters:
I20250901 01:53:22.587795 1686 raft_consensus.cc:2802] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:22.587947 1686 raft_consensus.cc:491] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:22.588063 1686 raft_consensus.cc:3058] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.588811 1713 tablet_bootstrap.cc:492] T de13eb65b0df4b8ebc049ff753123d53 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:22.588594 1023 raft_consensus.cc:2466] T 3f1db382c27e455cae9ddf28a79ada2b P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2ba2450480454388723cd729cc438e in term 1.
I20250901 01:53:22.588831 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "089fe953bba0403db288afefaf750c10" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:22.589797 1713 ts_tablet_manager.cc:1397] T de13eb65b0df4b8ebc049ff753123d53 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.010s
I20250901 01:53:22.590821 1713 raft_consensus.cc:357] T de13eb65b0df4b8ebc049ff753123d53 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.591346 1713 raft_consensus.cc:383] T de13eb65b0df4b8ebc049ff753123d53 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:22.591466 1713 raft_consensus.cc:738] T de13eb65b0df4b8ebc049ff753123d53 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
I20250901 01:53:22.591877 1713 consensus_queue.cc:260] T de13eb65b0df4b8ebc049ff753123d53 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.594043 1713 ts_tablet_manager.cc:1428] T de13eb65b0df4b8ebc049ff753123d53 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250901 01:53:22.595625 1760 ts_tablet_manager.cc:1918] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:53:22.595815 1760 log.cc:1199] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal/wals/138d19e14b734c1cace0d7ba51ab2caa
I20250901 01:53:22.596292 1760 ts_tablet_manager.cc:1939] T 138d19e14b734c1cace0d7ba51ab2caa P e3c774a4edf44d6098a2332e734967a5: Deleting consensus metadata
I20250901 01:53:22.596853 1761 ts_tablet_manager.cc:1918] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:53:22.597066 1761 log.cc:1199] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal/wals/138d19e14b734c1cace0d7ba51ab2caa
I20250901 01:53:22.597548 1761 ts_tablet_manager.cc:1939] T 138d19e14b734c1cace0d7ba51ab2caa P ad2ba2450480454388723cd729cc438e: Deleting consensus metadata
I20250901 01:53:22.597993 1686 raft_consensus.cc:513] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.598865 1686 leader_election.cc:290] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Requested vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:22.599205 1022 raft_consensus.cc:2391] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3c774a4edf44d6098a2332e734967a5 in current term 1: Already voted for candidate b65b3154aefd49f7a37f70e9207e9853 in this term.
I20250901 01:53:22.600195 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "089fe953bba0403db288afefaf750c10" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:22.600514 1305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "089fe953bba0403db288afefaf750c10" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:22.600769 1305 raft_consensus.cc:3058] T 089fe953bba0403db288afefaf750c10 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.603998 1669 raft_consensus.cc:513] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.604540 1166 raft_consensus.cc:2391] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b65b3154aefd49f7a37f70e9207e9853 in current term 1: Already voted for candidate e3c774a4edf44d6098a2332e734967a5 in this term.
I20250901 01:53:22.605345 1306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "089fe953bba0403db288afefaf750c10" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:22.605986 1761 tablet_replica.cc:331] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e: stopping tablet replica
I20250901 01:53:22.606257 1761 raft_consensus.cc:2241] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:53:22.606688 1761 raft_consensus.cc:2270] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:22.606869 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "089fe953bba0403db288afefaf750c10" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:22.607055 1760 tablet_replica.cc:331] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5: stopping tablet replica
I20250901 01:53:22.607120 1022 raft_consensus.cc:2391] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e3c774a4edf44d6098a2332e734967a5 in current term 1: Already voted for candidate b65b3154aefd49f7a37f70e9207e9853 in this term.
I20250901 01:53:22.607267 1760 raft_consensus.cc:2241] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:22.607378 1761 ts_tablet_manager.cc:1905] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:22.607496 1760 raft_consensus.cc:2270] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:22.607597 1103 leader_election.cc:304] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3c774a4edf44d6098a2332e734967a5; no voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853
I20250901 01:53:22.608129 1760 ts_tablet_manager.cc:1905] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:22.608588 1759 ts_tablet_manager.cc:1918] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:53:22.608749 1759 log.cc:1199] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/wals/138d19e14b734c1cace0d7ba51ab2caa
I20250901 01:53:22.609205 1759 ts_tablet_manager.cc:1939] T 138d19e14b734c1cace0d7ba51ab2caa P b65b3154aefd49f7a37f70e9207e9853: Deleting consensus metadata
I20250901 01:53:22.609638 1669 leader_election.cc:290] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Requested vote from peers ad2ba2450480454388723cd729cc438e (127.0.104.3:42927), b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:22.609864 1669 raft_consensus.cc:2747] T 089fe953bba0403db288afefaf750c10 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250901 01:53:22.641407 1305 raft_consensus.cc:2466] T 089fe953bba0403db288afefaf750c10 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:22.642084 954 leader_election.cc:304] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters: e3c774a4edf44d6098a2332e734967a5
I20250901 01:53:22.642704 1686 raft_consensus.cc:2802] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:22.642869 1686 raft_consensus.cc:695] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Becoming Leader. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Running, Role: LEADER
I20250901 01:53:22.643221 1686 consensus_queue.cc:237] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } } peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } }
I20250901 01:53:22.646296 1759 tablet_replica.cc:331] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853: stopping tablet replica
I20250901 01:53:22.646559 1759 raft_consensus.cc:2241] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:22.646801 1759 raft_consensus.cc:2270] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:22.647720 1759 ts_tablet_manager.cc:1905] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:22.649502 1795 raft_consensus.cc:491] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:22.649722 1795 raft_consensus.cc:513] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.651170 1795 leader_election.cc:290] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:22.651260 1305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de13eb65b0df4b8ebc049ff753123d53" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:22.651238 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de13eb65b0df4b8ebc049ff753123d53" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:22.651479 1305 raft_consensus.cc:2466] T de13eb65b0df4b8ebc049ff753123d53 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 0.
I20250901 01:53:22.651479 1022 raft_consensus.cc:2466] T de13eb65b0df4b8ebc049ff753123d53 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 0.
I20250901 01:53:22.651990 1101 leader_election.cc:304] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:22.652308 1795 raft_consensus.cc:2802] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:22.652451 1795 raft_consensus.cc:491] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:22.652565 1795 raft_consensus.cc:3058] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.688917 1794 consensus_queue.cc:1035] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:22.689829 1795 raft_consensus.cc:513] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.690542 1795 leader_election.cc:290] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Requested vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:22.691512 1761 ts_tablet_manager.cc:1918] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:53:22.691686 1761 log.cc:1199] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal/wals/9cd7993f6a99468caa9b13fc4e8c0ba1
I20250901 01:53:22.692181 1761 ts_tablet_manager.cc:1939] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P ad2ba2450480454388723cd729cc438e: Deleting consensus metadata
I20250901 01:53:22.692471 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de13eb65b0df4b8ebc049ff753123d53" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:22.692718 1022 raft_consensus.cc:3058] T de13eb65b0df4b8ebc049ff753123d53 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.693042 1306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de13eb65b0df4b8ebc049ff753123d53" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:22.693288 1306 raft_consensus.cc:3058] T de13eb65b0df4b8ebc049ff753123d53 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.695569 1760 ts_tablet_manager.cc:1918] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:53:22.696448 1760 log.cc:1199] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal/wals/9cd7993f6a99468caa9b13fc4e8c0ba1
I20250901 01:53:22.697041 1760 ts_tablet_manager.cc:1939] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P e3c774a4edf44d6098a2332e734967a5: Deleting consensus metadata
I20250901 01:53:22.706640 1794 consensus_queue.cc:1035] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:22.734264 1022 raft_consensus.cc:2466] T de13eb65b0df4b8ebc049ff753123d53 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 1.
I20250901 01:53:22.734783 1306 raft_consensus.cc:2466] T de13eb65b0df4b8ebc049ff753123d53 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 1.
I20250901 01:53:22.735244 1103 leader_election.cc:304] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:22.735837 1795 raft_consensus.cc:2802] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:22.736038 1795 raft_consensus.cc:695] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [term 1 LEADER]: Becoming Leader. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Running, Role: LEADER
I20250901 01:53:22.736346 1795 consensus_queue.cc:237] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.737655 1759 ts_tablet_manager.cc:1918] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:53:22.737865 1759 log.cc:1199] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/wals/9cd7993f6a99468caa9b13fc4e8c0ba1
I20250901 01:53:22.738822 1759 ts_tablet_manager.cc:1939] T 9cd7993f6a99468caa9b13fc4e8c0ba1 P b65b3154aefd49f7a37f70e9207e9853: Deleting consensus metadata
I20250901 01:53:22.755853 1795 raft_consensus.cc:491] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:22.756141 1795 raft_consensus.cc:513] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.756803 1795 leader_election.cc:290] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:22.757509 1306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2f0235fadf41ed8e705af5059c4702" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:22.757479 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2f0235fadf41ed8e705af5059c4702" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:22.757756 1306 raft_consensus.cc:2466] T 6b2f0235fadf41ed8e705af5059c4702 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 0.
I20250901 01:53:22.757776 1022 raft_consensus.cc:2466] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 0.
I20250901 01:53:22.758320 1103 leader_election.cc:304] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:22.758677 1795 raft_consensus.cc:2802] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:22.758793 1795 raft_consensus.cc:491] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:22.759074 1795 raft_consensus.cc:3058] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.788247 1795 raft_consensus.cc:513] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.789757 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2f0235fadf41ed8e705af5059c4702" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:22.789844 1306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2f0235fadf41ed8e705af5059c4702" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:22.790015 1022 raft_consensus.cc:3058] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.790095 1306 raft_consensus.cc:3058] T 6b2f0235fadf41ed8e705af5059c4702 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:22.790196 1795 leader_election.cc:290] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Requested vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
W20250901 01:53:22.797816 1100 connection.cc:537] server connection from 127.0.0.1:57118 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:22.797778 1241 connection.cc:537] server connection from 127.0.0.1:53910 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:22.797827 956 connection.cc:537] server connection from 127.0.0.1:41698 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:22.797885 1364 connection.cc:537] client connection to 127.0.104.61:33025 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:22.798406 1364 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.61:33025)
W20250901 01:53:22.798748 1364 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.61:33025)
W20250901 01:53:22.799558 1364 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.61:33025)
W20250901 01:53:22.800522 1364 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.61:33025)
I20250901 01:53:22.801785 658 master_runner.cc:186] Error getting master registration for 127.0.104.61:33025: Network error: recv got EOF from 127.0.104.61:33025 (error 108),
I20250901 01:53:22.801992 658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:53:22.802073 421 connection.cc:537] client connection to 127.0.104.61:33025 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:22.802526 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-task1fHXBS/build/asan/bin/kudu pid 725: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:53:22.802968 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1fHXBS/build/asan/bin/kudu with pid 725
I20250901 01:53:22.798976 1428 master_runner.cc:186] Error getting master registration for 127.0.104.61:33025: Network error: recv got EOF from 127.0.104.61:33025 (error 108),
I20250901 01:53:22.805094 1428 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:53:22.838914 1022 raft_consensus.cc:2466] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 1.
I20250901 01:53:22.839498 1306 raft_consensus.cc:2466] T 6b2f0235fadf41ed8e705af5059c4702 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 1.
I20250901 01:53:22.839862 1103 leader_election.cc:304] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:22.841224 1795 raft_consensus.cc:2802] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:22.841439 1795 raft_consensus.cc:695] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [term 1 LEADER]: Becoming Leader. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Running, Role: LEADER
I20250901 01:53:22.841754 1795 consensus_queue.cc:237] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:22.852826 1809 raft_consensus.cc:491] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b557230fc3b142c4bf5f6f58d241905f)
I20250901 01:53:22.853209 1809 raft_consensus.cc:513] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:22.854063 1809 leader_election.cc:290] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025)
I20250901 01:53:22.854919 698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "835a442fbc4c48eb9148ca677e410a59" candidate_term: 5 candidate_status { last_received { term: 4 index: 90 } } ignore_live_leader: false dest_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" is_pre_election: true
I20250901 01:53:22.855101 1810 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b557230fc3b142c4bf5f6f58d241905f)
I20250901 01:53:22.855307 1810 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:22.855697 698 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835a442fbc4c48eb9148ca677e410a59 in term 4.
W20250901 01:53:22.855628 1525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:33025: connect: Connection refused (error 111)
I20250901 01:53:22.856391 1519 leader_election.cc:304] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59; no voters:
I20250901 01:53:22.856698 1809 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250901 01:53:22.856817 1809 raft_consensus.cc:491] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b557230fc3b142c4bf5f6f58d241905f)
I20250901 01:53:22.856951 1809 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 4 FOLLOWER]: Advancing to term 5
I20250901 01:53:22.858769 1552 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 5 candidate_status { last_received { term: 4 index: 90 } } ignore_live_leader: false dest_uuid: "835a442fbc4c48eb9148ca677e410a59" is_pre_election: true
I20250901 01:53:22.859900 1810 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:22.859944 1525 leader_election.cc:336] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025): Network error: Client connection negotiation failed: client connection to 127.0.104.61:33025: connect: Connection refused (error 111)
W20250901 01:53:22.862600 675 leader_election.cc:336] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025): Network error: Client connection negotiation failed: client connection to 127.0.104.61:33025: connect: Connection refused (error 111)
I20250901 01:53:22.886885 1809 raft_consensus.cc:513] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:22.887512 1552 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2c35f5a36eeb4a93aecdd71e2d1c3a21 in current term 5: Already voted for candidate 835a442fbc4c48eb9148ca677e410a59 in this term.
I20250901 01:53:22.887871 1809 leader_election.cc:290] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [CANDIDATE]: Term 5 election: Requested vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025)
I20250901 01:53:22.888741 676 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21; no voters: 835a442fbc4c48eb9148ca677e410a59, b557230fc3b142c4bf5f6f58d241905f
I20250901 01:53:22.888883 698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "835a442fbc4c48eb9148ca677e410a59" candidate_term: 5 candidate_status { last_received { term: 4 index: 90 } } ignore_live_leader: false dest_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21"
I20250901 01:53:22.889113 698 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 4 FOLLOWER]: Advancing to term 5
W20250901 01:53:22.890415 1525 leader_election.cc:336] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025): Network error: Client connection negotiation failed: client connection to 127.0.104.61:33025: connect: Connection refused (error 111)
I20250901 01:53:22.910315 1794 consensus_queue.cc:1035] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:22.916930 1686 consensus_queue.cc:1035] T 3733efc4223d49a28f69cb598bdf7c94 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:22.987722 698 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835a442fbc4c48eb9148ca677e410a59 in term 5.
I20250901 01:53:22.988065 1810 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250901 01:53:22.988556 1519 leader_election.cc:304] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59; no voters: b557230fc3b142c4bf5f6f58d241905f
I20250901 01:53:22.989305 1809 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 5 FOLLOWER]: Leader election won for term 5
I20250901 01:53:22.991283 1809 raft_consensus.cc:695] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [term 5 LEADER]: Becoming Leader. State: Replica: 835a442fbc4c48eb9148ca677e410a59, State: Running, Role: LEADER
I20250901 01:53:22.991972 1809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 4.90, Last appended by leader: 90, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:22.994376 1818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 835a442fbc4c48eb9148ca677e410a59. Latest consensus state: current_term: 5 leader_uuid: "835a442fbc4c48eb9148ca677e410a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:22.994716 1818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:22.995612 1819 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 01:53:22.999181 1819 catalog_manager.cc:671] Loaded metadata for table default.table_bb66c7b604104254a48d435595b83fc4 [id=267f90dc52214e7986546d0e99a7f23a]
I20250901 01:53:22.999900 1819 catalog_manager.cc:671] Loaded metadata for table default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19]
I20250901 01:53:23.000454 1819 catalog_manager.cc:671] Loaded metadata for table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]
I20250901 01:53:23.000931 1819 catalog_manager.cc:671] Loaded metadata for table default.table_8082166db25745e78737b640063b3219 [id=37fc4ed840564a8780eeb9847b373712]
I20250901 01:53:23.001319 1819 catalog_manager.cc:671] Loaded metadata for table default.table_8f21a26e011c4ed1b1ae7cb364e09195 [id=39123cfe4f62420681f8d679d2d376b8]
I20250901 01:53:23.001685 1819 catalog_manager.cc:671] Loaded metadata for table default.table_9f3451ed25714464ad16b3e94214d0e3 [id=5575a859d470414e9a1bf99d5738fc3a]
I20250901 01:53:23.002256 1819 catalog_manager.cc:671] Loaded metadata for table default.table_da487bf0d57f439bb9da2ec57880a73b [id=7185f28e50df4fb9a4bf41cc5b73169d]
I20250901 01:53:23.002625 1819 catalog_manager.cc:671] Loaded metadata for table default.table_167322850c734684b896b875526a5c6e [id=7a69995eebe446629422a7fe0924e58a]
I20250901 01:53:23.003463 1819 catalog_manager.cc:671] Loaded metadata for table default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d]
I20250901 01:53:23.004812 1819 catalog_manager.cc:671] Loaded metadata for table default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98]
I20250901 01:53:23.005283 1819 catalog_manager.cc:671] Loaded metadata for table default.table_1ff7214e853748e58232b9bea0cb6912 [id=af5c975082de48cfa7295b2d8bd16635]
I20250901 01:53:23.005803 1819 catalog_manager.cc:671] Loaded metadata for table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647]
I20250901 01:53:23.006371 1819 catalog_manager.cc:671] Loaded metadata for table default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9]
I20250901 01:53:23.006928 1819 catalog_manager.cc:671] Loaded metadata for table default.table_facefcd0049d44a2b465d8315527aa08 [id=d0ce834bce8e4a3fa795fb9e3b6b06a7]
I20250901 01:53:23.007423 1819 catalog_manager.cc:671] Loaded metadata for table default.table_c95fbf8fa7a64aa18002f1e268cdea68 [id=d830537abb3c488ca7b925a8a491432a]
I20250901 01:53:23.008268 1819 catalog_manager.cc:671] Loaded metadata for table default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5]
I20250901 01:53:23.008829 1819 catalog_manager.cc:671] Loaded metadata for table default.table_99b272393a1b40b4ae2d3bbaedc9ea60 [id=f868207bffb24b6795d0f636974d231d]
I20250901 01:53:23.012358 1819 tablet_loader.cc:96] loaded metadata for tablet 089fe953bba0403db288afefaf750c10 (table default.table_99b272393a1b40b4ae2d3bbaedc9ea60 [id=f868207bffb24b6795d0f636974d231d])
I20250901 01:53:23.013314 1819 tablet_loader.cc:96] loaded metadata for tablet 3733efc4223d49a28f69cb598bdf7c94 (table default.table_facefcd0049d44a2b465d8315527aa08 [id=d0ce834bce8e4a3fa795fb9e3b6b06a7])
I20250901 01:53:23.013909 1819 tablet_loader.cc:96] loaded metadata for tablet 3f1db382c27e455cae9ddf28a79ada2b (table default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9])
I20250901 01:53:23.014396 1819 tablet_loader.cc:96] loaded metadata for tablet 4e1a5e7ec3f842ffa6f24e5b315eb6dd (table default.table_c95fbf8fa7a64aa18002f1e268cdea68 [id=d830537abb3c488ca7b925a8a491432a])
I20250901 01:53:23.014781 1819 tablet_loader.cc:96] loaded metadata for tablet 5274573577374b47b99fa310a5ca83bb (table default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19])
I20250901 01:53:23.015198 1819 tablet_loader.cc:96] loaded metadata for tablet 586fbc1ef054435aaaea9ae961f15115 (table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647])
I20250901 01:53:23.015708 1819 tablet_loader.cc:96] loaded metadata for tablet 6b2f0235fadf41ed8e705af5059c4702 (table default.table_167322850c734684b896b875526a5c6e [id=7a69995eebe446629422a7fe0924e58a])
I20250901 01:53:23.016204 1819 tablet_loader.cc:96] loaded metadata for tablet 8dd9f23bffa543a490c018108ad41051 (table default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98])
I20250901 01:53:23.016551 1819 tablet_loader.cc:96] loaded metadata for tablet 8f913036f34345fb8b20b3c9f431979e (table default.table_8f21a26e011c4ed1b1ae7cb364e09195 [id=39123cfe4f62420681f8d679d2d376b8])
I20250901 01:53:23.017400 1819 tablet_loader.cc:96] loaded metadata for tablet a6af5a07f3ca4fe4a8808b913551c091 (table default.table_1ff7214e853748e58232b9bea0cb6912 [id=af5c975082de48cfa7295b2d8bd16635])
I20250901 01:53:23.017939 1819 tablet_loader.cc:96] loaded metadata for tablet ad4ea5cf599f4719960fdf0b2e6fc054 (table default.table_9f3451ed25714464ad16b3e94214d0e3 [id=5575a859d470414e9a1bf99d5738fc3a])
I20250901 01:53:23.018311 1819 tablet_loader.cc:96] loaded metadata for tablet b302bfaed5104909a1d18f7529398348 (table default.table_8082166db25745e78737b640063b3219 [id=37fc4ed840564a8780eeb9847b373712])
I20250901 01:53:23.019376 1819 tablet_loader.cc:96] loaded metadata for tablet d258ddb89ab54c838f2fc8a08ab0f1a5 (table default.table_da487bf0d57f439bb9da2ec57880a73b [id=7185f28e50df4fb9a4bf41cc5b73169d])
I20250901 01:53:23.019716 1819 tablet_loader.cc:96] loaded metadata for tablet de13eb65b0df4b8ebc049ff753123d53 (table default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5])
I20250901 01:53:23.020143 1819 tablet_loader.cc:96] loaded metadata for tablet debd885f996642faa60cfa5af0283b5f (table default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d])
I20250901 01:53:23.020478 1819 tablet_loader.cc:96] loaded metadata for tablet e4df70f8191e4115991d42178cb67bd7 (table default.table_bb66c7b604104254a48d435595b83fc4 [id=267f90dc52214e7986546d0e99a7f23a])
I20250901 01:53:23.020787 1819 tablet_loader.cc:96] loaded metadata for tablet f81df1b890684c9cb01597c5bfebcefa (table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd])
I20250901 01:53:23.021088 1819 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 01:53:23.022567 1819 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:23.022656 1819 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 01:53:23.024346 1819 catalog_manager.cc:1506] Loading token signing keys...
I20250901 01:53:23.025624 1819 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: Loaded TSK: 0
I20250901 01:53:23.026930 1819 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 01:53:23.028240 1819 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 01:53:23.029528 1819 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 22
I20250901 01:53:23.039549 1534 master_service.cc:948] ReplaceTablet: received request to replace tablet 6b2f0235fadf41ed8e705af5059c4702 from {username='slave'} at 127.0.0.1:52854
I20250901 01:53:23.039513 1537 master_service.cc:948] ReplaceTablet: received request to replace tablet 3f1db382c27e455cae9ddf28a79ada2b from {username='slave'} at 127.0.0.1:52854
I20250901 01:53:23.039958 1538 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52854:
name: "default.Table_98a9d1400149426bab73e31aea263f5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:23.039405 1542 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52854:
name: "default.Table_56fde480a00a4e62b84eedba22b0bade"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:23.041023 1536 master_service.cc:948] ReplaceTablet: received request to replace tablet 4e1a5e7ec3f842ffa6f24e5b315eb6dd from {username='slave'} at 127.0.0.1:52854
I20250901 01:53:23.039834 1535 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52854:
name: "default.Table_0084f9e4b939410da1d13247891e6ac4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:23.041041 1533 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52854:
name: "default.Table_ed3c253535084a3399b30083bb388f72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:23.044615 1531 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52854:
name: "default.Table_f6c5cfd184ce474b883ce77bd225eb9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:23.045418 1530 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52854:
name: "default.Table_c1e876d5919f4c1f98a0ef9d1fcb9edc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:23.041394 1532 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52854:
name: "default.Table_cbe4bb8385f54f7f9359f495d8c663a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250901 01:53:23.051759 1638 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 5 out of 13 received events: NotificationEvent(eventId=23, eventTime=1756691601, eventType=CREATE_TABLE, dbName=default, tableName=table_98a9d1400149426bab73e31aea263f5b, message={"server":"","servicePrincipal":"","db":"default","table":"table_98a9d1400149426bab73e31aea263f5b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_98a9d1400149426bab73e31aea263f5b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691601},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_98a9d1400149426bab73e31aea263f5b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_98a9d1400149426bab73e31aea263f5b\",\"kudu.table_id\":\"49527ae170f04658b8b1eb01b07ffd04\",\"transient_lastDdlTime\":\"1756691601\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1756691601, eventType=ALTER_TABLE, dbName=default, tableName=table_757f6c2777014fe1b0686fc1405b4b5f, message={"server":"","servicePrincipal":"","db":"default","table":"table_61dfd07a8f784854b5d9e104e010daaa","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_61dfd07a8f784854b5d9e104e010daaa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691599},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_61dfd07a8f784854b5d9e104e010daaa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_61dfd07a8f784854b5d9e104e010daaa\",\"kudu.table_id\":\"ae0da2fb2d1542308ccb9a95247c1d98\",\"transient_lastDdlTime\":\"1756691599\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_757f6c2777014fe1b0686fc1405b4b5f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691599},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_757f6c2777014fe1b0686fc1405b4b5f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_757f6c2777014fe1b0686fc1405b4b5f\",\"kudu.table_id\":\"ae0da2fb2d1542308ccb9a95247c1d98\",\"transient_lastDdlTime\":\"1756691599\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_c2864548fe1a4719b73cf55d0933e538, message={"server":"","servicePrincipal":"","db":"default","table":"table_c2864548fe1a4719b73cf55d0933e538","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c2864548fe1a4719b73cf55d0933e538\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691598},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_c2864548fe1a4719b73cf55d0933e538\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c2864548fe1a4719b73cf55d0933e538\",\"kudu.table_id\":\"31d721fb8b204568a0190dda6def01cd\",\"transient_lastDdlTime\":\"1756691598\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_56fde480a00a4e62b84eedba22b0bade, message={"server":"","servicePrincipal":"","db":"default","table":"table_56fde480a00a4e62b84eedba22b0bade","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_56fde480a00a4e62b84eedba22b0bade\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691601},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_56fde480a00a4e62b84eedba22b0bade\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_56fde480a00a4e62b84eedba22b0bade\",\"kudu.table_id\":\"181eeac3e2394bcbaf3071e3fad223d8\",\"transient_lastDdlTime\":\"1756691601\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_7722e49197be4ca9b9c7e67052f07c4b, message={"server":"","servicePrincipal":"","db":"default","table":"table_7722e49197be4ca9b9c7e67052f07c4b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_7722e49197be4ca9b9c7e67052f07c4b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691598},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_7722e49197be4ca9b9c7e67052f07c4b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7722e49197be4ca9b9c7e67052f07c4b\",\"kudu.table_id\":\"2e950743029d48a49f05207cd5faee19\",\"transient_lastDdlTime\":\"1756691598\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
W20250901 01:53:23.052435 1525 consensus_peers.cc:489] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 -> Peer b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025): Couldn't send request to peer b557230fc3b142c4bf5f6f58d241905f. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:33025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7fcd1a285dcd google::LogMessage::Fail() at ??:0
I20250901 01:53:23.055117 698 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 5 FOLLOWER]: Refusing update from remote peer 835a442fbc4c48eb9148ca677e410a59: Log matching property violated. Preceding OpId in replica: term: 4 index: 90. Preceding OpId from leader: term: 5 index: 92. (index mismatch)
I20250901 01:53:23.056155 1818 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
@ 0x7fcd1a289b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fcd1a2857cc google::LogMessage::Flush() at ??:0
@ 0x7fcd1a285a11 google::LogMessage::~LogMessage() at ??:0
I20250901 01:53:23.064100 1810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 835a442fbc4c48eb9148ca677e410a59. Latest consensus state: current_term: 5 leader_uuid: "835a442fbc4c48eb9148ca677e410a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:23.064429 1810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:23.068476 1809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "835a442fbc4c48eb9148ca677e410a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:23.068893 1809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59 [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:23.069958 1534 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: Sending DeleteTablet for 3 replicas of tablet 6b2f0235fadf41ed8e705af5059c4702
I20250901 01:53:23.070358 1536 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 835a442fbc4c48eb9148ca677e410a59: Sending DeleteTablet for 3 replicas of tablet 4e1a5e7ec3f842ffa6f24e5b315eb6dd
@ 0x7fcd306ba794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250901 01:53:23.072335 1534 catalog_manager.cc:6652] ReplaceTablet: tablet 6b2f0235fadf41ed8e705af5059c4702 deleted and replaced by tablet 80d0e50666a34df39a4cc09977effe5c
I20250901 01:53:23.074503 1536 catalog_manager.cc:6652] ReplaceTablet: tablet 4e1a5e7ec3f842ffa6f24e5b315eb6dd deleted and replaced by tablet d8709bb983864d9f9e65eaf8f21a13f8
I20250901 01:53:23.082468 1534 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52854:
name: "default.Table_2368f4dfae714931b490e2544bb27666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:23.083532 1536 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52854:
table { table_name: "default.Table_facefcd0049d44a2b465d8315527aa08" } modify_external_catalogs: true
I20250901 01:53:23.083916 1536 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52854:
table { table_name: "default.Table_facefcd0049d44a2b465d8315527aa08" } modify_external_catalogs: true
@ 0x7fcd306b9041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fcd306c0cbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250901 01:53:23.094435 1810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "835a442fbc4c48eb9148ca677e410a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:23.094715 1810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fcd306c0b39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250901 01:53:23.105135 1003 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b2f0235fadf41ed8e705af5059c4702 with delete_type TABLET_DATA_DELETED (replaced by tablet 80d0e50666a34df39a4cc09977effe5c) from {username='slave'} at 127.0.0.1:41712
I20250901 01:53:23.105501 1004 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e1a5e7ec3f842ffa6f24e5b315eb6dd with delete_type TABLET_DATA_DELETED (replaced by tablet d8709bb983864d9f9e65eaf8f21a13f8) from {username='slave'} at 127.0.0.1:41712
I20250901 01:53:23.105764 1759 tablet_replica.cc:331] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853: stopping tablet replica
@ 0x7fcd1cfe2cc9 kudu::Thread::SuperviseThread() at ??:0
I20250901 01:53:23.106071 1759 raft_consensus.cc:2241] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:23.106325 1759 raft_consensus.cc:2270] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:23.107170 1759 ts_tablet_manager.cc:1905] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fcd204046db start_thread at ??:0
@ 0x7fcd1913671f clone at ??:0
I20250901 01:53:23.163537 1795 raft_consensus.cc:491] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:23.163810 1795 raft_consensus.cc:513] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:23.164537 1795 leader_election.cc:290] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:23.165225 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "debd885f996642faa60cfa5af0283b5f" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853" is_pre_election: true
I20250901 01:53:23.165280 1306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "debd885f996642faa60cfa5af0283b5f" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:23.165458 1022 raft_consensus.cc:2466] T debd885f996642faa60cfa5af0283b5f P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 0.
I20250901 01:53:23.165537 1306 raft_consensus.cc:2466] T debd885f996642faa60cfa5af0283b5f P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 0.
I20250901 01:53:23.165993 1103 leader_election.cc:304] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:23.166342 1795 raft_consensus.cc:2802] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:23.166482 1795 raft_consensus.cc:491] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:23.166570 1795 raft_consensus.cc:3058] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:23.191062 1795 raft_consensus.cc:513] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:23.192028 1795 leader_election.cc:290] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Requested vote from peers b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:23.193058 1022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "debd885f996642faa60cfa5af0283b5f" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65b3154aefd49f7a37f70e9207e9853"
I20250901 01:53:23.193308 1022 raft_consensus.cc:3058] T debd885f996642faa60cfa5af0283b5f P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:23.195837 1306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "debd885f996642faa60cfa5af0283b5f" candidate_uuid: "e3c774a4edf44d6098a2332e734967a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:23.196133 1306 raft_consensus.cc:3058] T debd885f996642faa60cfa5af0283b5f P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:23.196516 1759 ts_tablet_manager.cc:1918] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:53:23.196645 1759 log.cc:1199] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/wals/6b2f0235fadf41ed8e705af5059c4702
I20250901 01:53:23.196926 1759 ts_tablet_manager.cc:1939] T 6b2f0235fadf41ed8e705af5059c4702 P b65b3154aefd49f7a37f70e9207e9853: Deleting consensus metadata
I20250901 01:53:23.230571 1718 consensus_queue.cc:1035] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:23.238229 1765 consensus_queue.cc:1035] T ad4ea5cf599f4719960fdf0b2e6fc054 P ad2ba2450480454388723cd729cc438e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:23.281549 1306 raft_consensus.cc:2466] T debd885f996642faa60cfa5af0283b5f P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 1.
I20250901 01:53:23.281576 1022 raft_consensus.cc:2466] T debd885f996642faa60cfa5af0283b5f P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c774a4edf44d6098a2332e734967a5 in term 1.
I20250901 01:53:23.282298 1101 leader_election.cc:304] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:23.282693 1795 raft_consensus.cc:2802] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:23.282855 1795 raft_consensus.cc:695] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [term 1 LEADER]: Becoming Leader. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Running, Role: LEADER
I20250901 01:53:23.283186 1759 tablet_replica.cc:331] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853: stopping tablet replica
I20250901 01:53:23.283161 1795 consensus_queue.cc:237] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:23.283459 1759 raft_consensus.cc:2241] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:23.283669 1759 raft_consensus.cc:2270] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:23.284345 1759 ts_tablet_manager.cc:1905] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:23.293056 1765 consensus_queue.cc:1035] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:23.300148 1718 consensus_queue.cc:1035] T b302bfaed5104909a1d18f7529398348 P ad2ba2450480454388723cd729cc438e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:53:23.361588 956 connection.cc:537] client connection to 127.0.104.60:36985 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:23.362497 1830 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 01:53:23.073548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.0.104.3:42927)
0901 01:53:23.110133 (+ 36585us) server_negotiation.cc:206] Beginning negotiation
0901 01:53:23.110138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 01:53:23.110190 (+ 52us) server_negotiation.cc:407] Connection header received
0901 01:53:23.110269 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 01:53:23.110273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 01:53:23.110345 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 01:53:23.110455 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 01:53:23.361364 (+250909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53914: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":36445,"thread_start_us":37118,"threads_started":1}
W20250901 01:53:23.362843 1365 connection.cc:537] client connection to 127.0.104.60:36985 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:23.363976 676 connection.cc:537] server connection from 127.0.0.1:58186 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:23.366603 1832 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 01:53:23.073836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.0.104.2:35245)
0901 01:53:23.110099 (+ 36263us) server_negotiation.cc:206] Beginning negotiation
0901 01:53:23.110106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 01:53:23.110134 (+ 28us) server_negotiation.cc:407] Connection header received
0901 01:53:23.110259 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 01:53:23.110268 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 01:53:23.110358 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 01:53:23.110474 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 01:53:23.365526 (+255052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57128: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":36101,"thread_start_us":37010,"threads_started":1}
W20250901 01:53:23.367105 955 connection.cc:537] server connection from 127.0.0.1:41712 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:23.367377 1243 connection.cc:537] client connection to 127.0.104.60:36985 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:23.367694 1103 connection.cc:537] client connection to 127.0.104.60:36985 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:53:23.384783 1759 ts_tablet_manager.cc:1918] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:23.385012 1759 log.cc:1199] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/wals/4e1a5e7ec3f842ffa6f24e5b315eb6dd
I20250901 01:53:23.385505 1759 ts_tablet_manager.cc:1939] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P b65b3154aefd49f7a37f70e9207e9853: Deleting consensus metadata
I20250901 01:53:23.402202 1718 consensus_queue.cc:1035] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:23.412813 1718 consensus_queue.cc:1035] T 3f1db382c27e455cae9ddf28a79ada2b P ad2ba2450480454388723cd729cc438e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:23.443133 1766 consensus_queue.cc:1035] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:23.449609 1849 consensus_queue.cc:1035] T e4df70f8191e4115991d42178cb67bd7 P ad2ba2450480454388723cd729cc438e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:53:23.457223 955 connection.cc:441] server connection from 127.0.0.1:41712 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41712 (request call id 1) could send its response
I20250901 01:53:23.568655 1810 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 835a442fbc4c48eb9148ca677e410a59)
I20250901 01:53:23.568894 1810 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:23.569690 1810 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:23.572801 675 leader_election.cc:336] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025): Network error: Client connection negotiation failed: client connection to 127.0.104.61:33025: connect: Connection refused (error 111)
W20250901 01:53:23.573060 676 leader_election.cc:336] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:23.573207 676 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21; no voters: 835a442fbc4c48eb9148ca677e410a59, b557230fc3b142c4bf5f6f58d241905f
I20250901 01:53:23.573541 1810 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250901 01:53:23.599156 1795 consensus_queue.cc:1035] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:23.605624 1795 consensus_queue.cc:1035] T de13eb65b0df4b8ebc049ff753123d53 P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:23.663133 1686 consensus_queue.cc:1035] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:23.670508 1799 consensus_queue.cc:1035] T 089fe953bba0403db288afefaf750c10 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:23.765998 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1fHXBS/build/asan/bin/kudu
/tmp/dist-test-task1fHXBS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:33025
--webserver_interface=127.0.104.61
--webserver_port=34857
--builtin_ntp_servers=127.0.104.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250901 01:53:24.061748 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 1: this message will repeat every 5th retry.
W20250901 01:53:24.071110 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:53:24.084039 1867 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 835a442fbc4c48eb9148ca677e410a59)
I20250901 01:53:24.084239 1867 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:24.085767 1795 consensus_queue.cc:1035] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:24.086519 1867 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:24.087133 676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20250901 01:53:24.092576 676 leader_election.cc:336] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:53:24.092880 675 leader_election.cc:336] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025): Network error: Client connection negotiation failed: client connection to 127.0.104.61:33025: connect: Connection refused (error 111)
I20250901 01:53:24.093139 675 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21; no voters: 835a442fbc4c48eb9148ca677e410a59, b557230fc3b142c4bf5f6f58d241905f
I20250901 01:53:24.093685 1867 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250901 01:53:24.157402 1866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:53:24.157774 1866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:53:24.157891 1866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250901 01:53:24.163108 1795 consensus_queue.cc:1035] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:53:24.168861 1866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 01:53:24.169073 1866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:53:24.169195 1866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 01:53:24.169270 1866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 01:53:24.171232 1859 consensus_queue.cc:1035] T debd885f996642faa60cfa5af0283b5f P e3c774a4edf44d6098a2332e734967a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:24.184130 1866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=34857
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
I20250901 01:53:24.185824 1866 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:53:24.187700 1866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 01:53:24.196599 1882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:24.196544 1881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:24.196568 1884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:24.197402 1866 server_base.cc:1047] running on GCE node
I20250901 01:53:24.198320 1866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:53:24.200080 1866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:53:24.201363 1866 hybrid_clock.cc:648] HybridClock initialized: now 1756691604201303 us; error 59 us; skew 500 ppm
I20250901 01:53:24.201831 1866 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:53:24.204427 1866 webserver.cc:480] Webserver started at http://127.0.104.61:34857/ using document root <none> and password file <none>
I20250901 01:53:24.205282 1866 fs_manager.cc:362] Metadata directory not provided
I20250901 01:53:24.205418 1866 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 01:53:24.211773 1866 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20250901 01:53:24.215901 1890 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:24.217767 1866 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250901 01:53:24.217949 1866 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data,/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
uuid: "b557230fc3b142c4bf5f6f58d241905f"
format_stamp: "Formatted at 2025-09-01 01:53:13 on dist-test-slave-r4vn"
I20250901 01:53:24.218765 1866 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:53:24.256820 1866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 01:53:24.257689 1866 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:53:24.257969 1866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:53:24.278970 1866 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:33025
I20250901 01:53:24.278970 1941 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:33025 every 8 connection(s)
I20250901 01:53:24.281307 1866 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/info.pb
I20250901 01:53:24.288395 1942 sys_catalog.cc:263] Verifying existing consensus state
I20250901 01:53:24.288442 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1fHXBS/build/asan/bin/kudu as pid 1866
I20250901 01:53:24.292937 1942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Bootstrap starting.
I20250901 01:53:24.325896 1942 log.cc:826] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Log is configured to *not* fsync() on all Append() calls
I20250901 01:53:24.327090 1354 heartbeater.cc:344] Connected to a master server at 127.0.104.61:33025
I20250901 01:53:24.397647 1942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Bootstrap replayed 1/1 log segments. Stats: ops{read=90 overwritten=0 applied=90 ignored=0} inserts{seen=60 ignored=0} mutations{seen=68 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 01:53:24.398344 1942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Bootstrap complete.
I20250901 01:53:24.404784 1942 raft_consensus.cc:357] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:24.405575 1942 raft_consensus.cc:738] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: b557230fc3b142c4bf5f6f58d241905f, State: Initialized, Role: FOLLOWER
I20250901 01:53:24.406384 1942 consensus_queue.cc:260] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 90, Last appended: 4.90, Last appended by leader: 90, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:24.407403 1949 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:24.407740 1949 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:24.408545 1942 sys_catalog.cc:564] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: configured and running, proceeding with master startup.
I20250901 01:53:24.437441 1963 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:24.437575 1963 catalog_manager.cc:1554] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: loading cluster ID for follower catalog manager: success
I20250901 01:53:24.442970 1963 catalog_manager.cc:1576] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: acquiring CA information for follower catalog manager: success
I20250901 01:53:24.449690 1963 catalog_manager.cc:1604] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 01:53:24.469798 1907 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ad2ba2450480454388723cd729cc438e" instance_seqno: 1756691596941203) as {username='slave'} at 127.0.104.3:44349; Asking this server to re-register.
I20250901 01:53:24.470841 1354 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:24.475343 1906 ts_manager.cc:194] Registered new tserver with Master: ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:24.475438 1071 heartbeater.cc:344] Connected to a master server at 127.0.104.61:33025
I20250901 01:53:24.479656 1906 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" instance_seqno: 1756691595802033) as {username='slave'} at 127.0.104.1:59681; Asking this server to re-register.
I20250901 01:53:24.476550 1866 master_runner.cc:186] Error getting master registration for 127.0.104.60:36985: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111),
I20250901 01:53:24.480636 1071 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:24.480813 1866 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:53:24.482565 1906 ts_manager.cc:194] Registered new tserver with Master: b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:24.604058 1214 heartbeater.cc:344] Connected to a master server at 127.0.104.61:33025
I20250901 01:53:24.606992 1906 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" instance_seqno: 1756691596380664) as {username='slave'} at 127.0.104.2:53405; Asking this server to re-register.
I20250901 01:53:24.607721 1214 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:24.609591 1906 ts_manager.cc:194] Registered new tserver with Master: e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:24.956728 1971 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:24.957010 1971 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:24.958918 1971 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:24.959498 698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b557230fc3b142c4bf5f6f58d241905f" candidate_term: 5 candidate_status { last_received { term: 4 index: 90 } } ignore_live_leader: false dest_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" is_pre_election: true
W20250901 01:53:24.959643 1894 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:24.959833 698 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b557230fc3b142c4bf5f6f58d241905f in current term 5: Already voted for candidate 835a442fbc4c48eb9148ca677e410a59 in this term.
W20250901 01:53:24.961248 1894 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:24.961390 1894 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:24.961784 1971 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 4 FOLLOWER]: Advancing to term 5
I20250901 01:53:24.965533 1971 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250901 01:53:25.048843 1973 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 835a442fbc4c48eb9148ca677e410a59)
I20250901 01:53:25.049057 1973 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:25.049770 1973 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:25.052253 676 leader_election.cc:336] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:25.056212 1917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 6 candidate_status { last_received { term: 5 index: 94 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f" is_pre_election: true
I20250901 01:53:25.056644 1917 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c35f5a36eeb4a93aecdd71e2d1c3a21 in term 5.
I20250901 01:53:25.057109 675 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, b557230fc3b142c4bf5f6f58d241905f; no voters: 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:25.057420 1973 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250901 01:53:25.057564 1973 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 835a442fbc4c48eb9148ca677e410a59)
I20250901 01:53:25.057644 1973 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 5 FOLLOWER]: Advancing to term 6
I20250901 01:53:25.059574 1973 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:25.060139 1973 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 6 election: Requested vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:25.060689 1917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 6 candidate_status { last_received { term: 5 index: 94 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f"
I20250901 01:53:25.060905 1917 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 5 FOLLOWER]: Advancing to term 6
W20250901 01:53:25.061637 676 leader_election.cc:336] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:25.063321 1917 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c35f5a36eeb4a93aecdd71e2d1c3a21 in term 6.
I20250901 01:53:25.063841 675 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, b557230fc3b142c4bf5f6f58d241905f; no voters: 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:25.064139 1973 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 6 FOLLOWER]: Leader election won for term 6
I20250901 01:53:25.065057 1973 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 6 LEADER]: Becoming Leader. State: Replica: 2c35f5a36eeb4a93aecdd71e2d1c3a21, State: Running, Role: LEADER
I20250901 01:53:25.065414 1973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 5.94, Last appended by leader: 94, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:25.067656 1977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c35f5a36eeb4a93aecdd71e2d1c3a21. Latest consensus state: current_term: 6 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:25.067956 1977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:25.478690 1917 raft_consensus.cc:1273] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Refusing update from remote peer 2c35f5a36eeb4a93aecdd71e2d1c3a21: Log matching property violated. Preceding OpId in replica: term: 4 index: 90. Preceding OpId from leader: term: 6 index: 95. (index mismatch)
I20250901 01:53:25.479437 1977 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 90, Time since last communication: 0.000s
I20250901 01:53:25.486599 1986 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:25.486996 1986 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:25.488183 1983 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c35f5a36eeb4a93aecdd71e2d1c3a21. Latest consensus state: current_term: 6 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:25.488471 1983 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:25.490556 1977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:25.490805 1977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:25.491267 1983 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:25.491479 1983 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
W20250901 01:53:25.492087 676 consensus_peers.cc:489] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 -> Peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Couldn't send request to peer 835a442fbc4c48eb9148ca677e410a59. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:53:25.585379 1979 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 01:53:25.588161 1979 catalog_manager.cc:671] Loaded metadata for table default.table_bb66c7b604104254a48d435595b83fc4 [id=267f90dc52214e7986546d0e99a7f23a]
I20250901 01:53:25.588613 1979 catalog_manager.cc:671] Loaded metadata for table default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19]
I20250901 01:53:25.588964 1979 catalog_manager.cc:671] Loaded metadata for table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]
I20250901 01:53:25.589319 1979 catalog_manager.cc:671] Loaded metadata for table default.table_8082166db25745e78737b640063b3219 [id=37fc4ed840564a8780eeb9847b373712]
I20250901 01:53:25.589643 1979 catalog_manager.cc:671] Loaded metadata for table default.table_8f21a26e011c4ed1b1ae7cb364e09195 [id=39123cfe4f62420681f8d679d2d376b8]
I20250901 01:53:25.590067 1979 catalog_manager.cc:671] Loaded metadata for table default.table_9f3451ed25714464ad16b3e94214d0e3 [id=5575a859d470414e9a1bf99d5738fc3a]
I20250901 01:53:25.590627 1979 catalog_manager.cc:671] Loaded metadata for table default.table_da487bf0d57f439bb9da2ec57880a73b [id=7185f28e50df4fb9a4bf41cc5b73169d]
I20250901 01:53:25.590983 1979 catalog_manager.cc:671] Loaded metadata for table default.table_167322850c734684b896b875526a5c6e [id=7a69995eebe446629422a7fe0924e58a]
I20250901 01:53:25.591285 1979 catalog_manager.cc:671] Loaded metadata for table default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d]
I20250901 01:53:25.591601 1979 catalog_manager.cc:671] Loaded metadata for table default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98]
I20250901 01:53:25.591900 1979 catalog_manager.cc:671] Loaded metadata for table default.table_1ff7214e853748e58232b9bea0cb6912 [id=af5c975082de48cfa7295b2d8bd16635]
I20250901 01:53:25.592221 1979 catalog_manager.cc:671] Loaded metadata for table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647]
I20250901 01:53:25.592516 1979 catalog_manager.cc:671] Loaded metadata for table default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9]
I20250901 01:53:25.592892 1979 catalog_manager.cc:671] Loaded metadata for table default.table_facefcd0049d44a2b465d8315527aa08 [id=d0ce834bce8e4a3fa795fb9e3b6b06a7]
I20250901 01:53:25.593211 1979 catalog_manager.cc:671] Loaded metadata for table default.table_c95fbf8fa7a64aa18002f1e268cdea68 [id=d830537abb3c488ca7b925a8a491432a]
I20250901 01:53:25.593677 1979 catalog_manager.cc:671] Loaded metadata for table default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5]
I20250901 01:53:25.594002 1979 catalog_manager.cc:671] Loaded metadata for table default.table_99b272393a1b40b4ae2d3bbaedc9ea60 [id=f868207bffb24b6795d0f636974d231d]
I20250901 01:53:25.595724 1979 tablet_loader.cc:96] loaded metadata for tablet 089fe953bba0403db288afefaf750c10 (table default.table_99b272393a1b40b4ae2d3bbaedc9ea60 [id=f868207bffb24b6795d0f636974d231d])
I20250901 01:53:25.596239 1979 tablet_loader.cc:96] loaded metadata for tablet 3733efc4223d49a28f69cb598bdf7c94 (table default.table_facefcd0049d44a2b465d8315527aa08 [id=d0ce834bce8e4a3fa795fb9e3b6b06a7])
I20250901 01:53:25.596854 1979 tablet_loader.cc:96] loaded metadata for tablet 5274573577374b47b99fa310a5ca83bb (table default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19])
I20250901 01:53:25.597190 1979 tablet_loader.cc:96] loaded metadata for tablet 586fbc1ef054435aaaea9ae961f15115 (table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647])
I20250901 01:53:25.597862 1979 tablet_loader.cc:96] loaded metadata for tablet 80d0e50666a34df39a4cc09977effe5c (table default.table_167322850c734684b896b875526a5c6e [id=7a69995eebe446629422a7fe0924e58a])
I20250901 01:53:25.598305 1979 tablet_loader.cc:96] loaded metadata for tablet 8dd9f23bffa543a490c018108ad41051 (table default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98])
I20250901 01:53:25.598584 1979 tablet_loader.cc:96] loaded metadata for tablet 8f913036f34345fb8b20b3c9f431979e (table default.table_8f21a26e011c4ed1b1ae7cb364e09195 [id=39123cfe4f62420681f8d679d2d376b8])
I20250901 01:53:25.599256 1979 tablet_loader.cc:96] loaded metadata for tablet a6af5a07f3ca4fe4a8808b913551c091 (table default.table_1ff7214e853748e58232b9bea0cb6912 [id=af5c975082de48cfa7295b2d8bd16635])
I20250901 01:53:25.599736 1979 tablet_loader.cc:96] loaded metadata for tablet ad4ea5cf599f4719960fdf0b2e6fc054 (table default.table_9f3451ed25714464ad16b3e94214d0e3 [id=5575a859d470414e9a1bf99d5738fc3a])
I20250901 01:53:25.599993 1979 tablet_loader.cc:96] loaded metadata for tablet b0064e7090a54fb18ee9a761ddec2c83 (table default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9])
I20250901 01:53:25.600267 1979 tablet_loader.cc:96] loaded metadata for tablet b302bfaed5104909a1d18f7529398348 (table default.table_8082166db25745e78737b640063b3219 [id=37fc4ed840564a8780eeb9847b373712])
I20250901 01:53:25.601137 1979 tablet_loader.cc:96] loaded metadata for tablet d258ddb89ab54c838f2fc8a08ab0f1a5 (table default.table_da487bf0d57f439bb9da2ec57880a73b [id=7185f28e50df4fb9a4bf41cc5b73169d])
I20250901 01:53:25.601394 1979 tablet_loader.cc:96] loaded metadata for tablet d8709bb983864d9f9e65eaf8f21a13f8 (table default.table_c95fbf8fa7a64aa18002f1e268cdea68 [id=d830537abb3c488ca7b925a8a491432a])
I20250901 01:53:25.601671 1979 tablet_loader.cc:96] loaded metadata for tablet de13eb65b0df4b8ebc049ff753123d53 (table default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5])
I20250901 01:53:25.601966 1979 tablet_loader.cc:96] loaded metadata for tablet debd885f996642faa60cfa5af0283b5f (table default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d])
I20250901 01:53:25.602276 1979 tablet_loader.cc:96] loaded metadata for tablet e4df70f8191e4115991d42178cb67bd7 (table default.table_bb66c7b604104254a48d435595b83fc4 [id=267f90dc52214e7986546d0e99a7f23a])
I20250901 01:53:25.602571 1979 tablet_loader.cc:96] loaded metadata for tablet f81df1b890684c9cb01597c5bfebcefa (table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd])
I20250901 01:53:25.603066 1979 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 01:53:25.604339 1979 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:25.604425 1979 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 01:53:25.605952 1979 catalog_manager.cc:1506] Loading token signing keys...
I20250901 01:53:25.607174 1979 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Loaded TSK: 0
I20250901 01:53:25.607604 1979 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 01:53:25.608688 1979 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 01:53:25.609885 1979 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 22
F20250901 01:53:25.626462 905 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 5 out of 13 received events: NotificationEvent(eventId=23, eventTime=1756691601, eventType=CREATE_TABLE, dbName=default, tableName=table_98a9d1400149426bab73e31aea263f5b, message={"server":"","servicePrincipal":"","db":"default","table":"table_98a9d1400149426bab73e31aea263f5b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_98a9d1400149426bab73e31aea263f5b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691601},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_98a9d1400149426bab73e31aea263f5b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_98a9d1400149426bab73e31aea263f5b\",\"kudu.table_id\":\"49527ae170f04658b8b1eb01b07ffd04\",\"transient_lastDdlTime\":\"1756691601\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1756691601, eventType=ALTER_TABLE, dbName=default, tableName=table_757f6c2777014fe1b0686fc1405b4b5f, message={"server":"","servicePrincipal":"","db":"default","table":"table_61dfd07a8f784854b5d9e104e010daaa","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_61dfd07a8f784854b5d9e104e010daaa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691599},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_61dfd07a8f784854b5d9e104e010daaa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_61dfd07a8f784854b5d9e104e010daaa\",\"kudu.table_id\":\"ae0da2fb2d1542308ccb9a95247c1d98\",\"transient_lastDdlTime\":\"1756691599\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_757f6c2777014fe1b0686fc1405b4b5f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691599},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_757f6c2777014fe1b0686fc1405b4b5f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_757f6c2777014fe1b0686fc1405b4b5f\",\"kudu.table_id\":\"ae0da2fb2d1542308ccb9a95247c1d98\",\"transient_lastDdlTime\":\"1756691599\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_c2864548fe1a4719b73cf55d0933e538, message={"server":"","servicePrincipal":"","db":"default","table":"table_c2864548fe1a4719b73cf55d0933e538","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c2864548fe1a4719b73cf55d0933e538\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691598},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_c2864548fe1a4719b73cf55d0933e538\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c2864548fe1a4719b73cf55d0933e538\",\"kudu.table_id\":\"31d721fb8b204568a0190dda6def01cd\",\"transient_lastDdlTime\":\"1756691598\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_56fde480a00a4e62b84eedba22b0bade, message={"server":"","servicePrincipal":"","db":"default","table":"table_56fde480a00a4e62b84eedba22b0bade","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_56fde480a00a4e62b84eedba22b0bade\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691601},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_56fde480a00a4e62b84eedba22b0bade\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_56fde480a00a4e62b84eedba22b0bade\",\"kudu.table_id\":\"181eeac3e2394bcbaf3071e3fad223d8\",\"transient_lastDdlTime\":\"1756691601\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_7722e49197be4ca9b9c7e67052f07c4b, message={"server":"","servicePrincipal":"","db":"default","table":"table_7722e49197be4ca9b9c7e67052f07c4b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_7722e49197be4ca9b9c7e67052f07c4b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691598},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_7722e49197be4ca9b9c7e67052f07c4b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7722e49197be4ca9b9c7e67052f07c4b\",\"kudu.table_id\":\"2e950743029d48a49f05207cd5faee19\",\"transient_lastDdlTime\":\"1756691598\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fce77adedcd google::LogMessage::Fail() at ??:0
@ 0x7fce77ae2b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fce77ade7cc google::LogMessage::Flush() at ??:0
@ 0x7fce77adea11 google::LogMessage::~LogMessage() at ??:0
I20250901 01:53:25.634078 689 master_service.cc:948] ReplaceTablet: received request to replace tablet 3f1db382c27e455cae9ddf28a79ada2b from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:25.634449 683 master_service.cc:948] ReplaceTablet: received request to replace tablet d258ddb89ab54c838f2fc8a08ab0f1a5 from {username='slave'} at 127.0.0.1:58182
I20250901 01:53:25.638741 689 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_c95fbf8fa7a64aa18002f1e268cdea68" } new_table_name: "default.Table_9afa854fa1c148929b0ede8977435270" modify_external_catalogs: true
I20250901 01:53:25.639075 682 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_5d5e7ea4ac7d4e5f9de3237058d11e80" } modify_external_catalogs: true
I20250901 01:53:25.639307 682 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_5d5e7ea4ac7d4e5f9de3237058d11e80" } modify_external_catalogs: true
I20250901 01:53:25.639379 684 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_61dfd07a8f784854b5d9e104e010daaa" } new_table_name: "default.Table_757f6c2777014fe1b0686fc1405b4b5f" modify_external_catalogs: true
@ 0x7fce8df13794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250901 01:53:25.640084 680 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_d1c05496169b472fa9b021013c4afd01" } modify_external_catalogs: true
I20250901 01:53:25.640285 680 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_d1c05496169b472fa9b021013c4afd01" } modify_external_catalogs: true
I20250901 01:53:25.640369 681 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_7e006da3e6d24d309af545afeac11ca4" } new_table_name: "default.Table_746437fc94f24e50977140cddc823eaf" modify_external_catalogs: true
I20250901 01:53:25.642026 687 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_1ff7214e853748e58232b9bea0cb6912" } new_table_name: "default.Table_1b2b89cc1bc2459b80fd9de967ebd9a1" modify_external_catalogs: true
I20250901 01:53:25.642473 686 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_da487bf0d57f439bb9da2ec57880a73b" } new_table_name: "default.Table_215810bf4c304b3cbce0d26ae7cfb7a0" modify_external_catalogs: true
I20250901 01:53:25.643498 688 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_7722e49197be4ca9b9c7e67052f07c4b" } modify_external_catalogs: true
I20250901 01:53:25.643735 688 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_7722e49197be4ca9b9c7e67052f07c4b" } modify_external_catalogs: true
I20250901 01:53:25.644055 685 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_8f21a26e011c4ed1b1ae7cb364e09195" } modify_external_catalogs: true
I20250901 01:53:25.644224 685 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58182:
table { table_name: "default.Table_8f21a26e011c4ed1b1ae7cb364e09195" } modify_external_catalogs: true
@ 0x7fce8df12041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250901 01:53:25.647529 683 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Sending DeleteTablet for 3 replicas of tablet d258ddb89ab54c838f2fc8a08ab0f1a5
I20250901 01:53:25.648130 683 catalog_manager.cc:6652] ReplaceTablet: tablet d258ddb89ab54c838f2fc8a08ab0f1a5 deleted and replaced by tablet 0e88f2701be74589b1337bd6f4ef87db
I20250901 01:53:25.649379 1286 tablet_service.cc:1515] Processing DeleteTablet for tablet d258ddb89ab54c838f2fc8a08ab0f1a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0e88f2701be74589b1337bd6f4ef87db) from {username='slave'} at 127.0.0.1:58732
I20250901 01:53:25.649461 1146 tablet_service.cc:1515] Processing DeleteTablet for tablet d258ddb89ab54c838f2fc8a08ab0f1a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0e88f2701be74589b1337bd6f4ef87db) from {username='slave'} at 127.0.0.1:53914
@ 0x7fce8df19cbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fce8df19b39 std::_Function_handler<>::_M_invoke() at ??:0
I20250901 01:53:25.654969 1997 tablet_replica.cc:331] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5: stopping tablet replica
I20250901 01:53:25.655105 1996 tablet_replica.cc:331] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e: stopping tablet replica
I20250901 01:53:25.655200 1997 raft_consensus.cc:2241] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:25.655313 1996 raft_consensus.cc:2241] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:53:25.655423 1997 raft_consensus.cc:2270] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:25.655680 1996 raft_consensus.cc:2270] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:25.650775 683 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58182:
name: "default.Table_0084f9e4b939410da1d13247891e6ac4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:25.656038 1997 ts_tablet_manager.cc:1905] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:25.656275 1996 ts_tablet_manager.cc:1905] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0xa774dd std::function<>::operator()() at ??:?
I20250901 01:53:25.656994 1004 tablet_service.cc:1515] Processing DeleteTablet for tablet d258ddb89ab54c838f2fc8a08ab0f1a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0e88f2701be74589b1337bd6f4ef87db) from {username='slave'} at 127.0.0.1:35622
@ 0x7fce7a83bcc9 kudu::Thread::SuperviseThread() at ??:0
I20250901 01:53:25.662824 1997 ts_tablet_manager.cc:1918] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:25.662971 1997 log.cc:1199] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal/wals/d258ddb89ab54c838f2fc8a08ab0f1a5
@ 0x7fce7dc5d6db start_thread at ??:0
@ 0x7fce7698f71f clone at ??:0
I20250901 01:53:25.669965 1997 ts_tablet_manager.cc:1939] T d258ddb89ab54c838f2fc8a08ab0f1a5 P e3c774a4edf44d6098a2332e734967a5: Deleting consensus metadata
I20250901 01:53:25.669965 1996 ts_tablet_manager.cc:1918] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:25.670095 1996 log.cc:1199] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal/wals/d258ddb89ab54c838f2fc8a08ab0f1a5
I20250901 01:53:25.670214 1999 tablet_replica.cc:331] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853: stopping tablet replica
I20250901 01:53:25.670468 1999 raft_consensus.cc:2241] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:25.670601 1996 ts_tablet_manager.cc:1939] T d258ddb89ab54c838f2fc8a08ab0f1a5 P ad2ba2450480454388723cd729cc438e: Deleting consensus metadata
I20250901 01:53:25.670684 1999 raft_consensus.cc:2270] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:25.671175 1999 ts_tablet_manager.cc:1905] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:25.678486 1999 ts_tablet_manager.cc:1918] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:25.678656 1999 log.cc:1199] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/wals/d258ddb89ab54c838f2fc8a08ab0f1a5
I20250901 01:53:25.679085 1999 ts_tablet_manager.cc:1939] T d258ddb89ab54c838f2fc8a08ab0f1a5 P b65b3154aefd49f7a37f70e9207e9853: Deleting consensus metadata
W20250901 01:53:25.847457 1100 connection.cc:537] client connection to 127.0.104.62:35805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:25.847481 956 connection.cc:537] server connection from 127.0.0.1:35622 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:25.847489 1362 connection.cc:537] client connection to 127.0.104.62:35805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:25.848134 1362 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.62:35805)
W20250901 01:53:25.848142 1103 connection.cc:537] server connection from 127.0.0.1:53914 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:25.848393 1362 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.62:35805)
W20250901 01:53:25.848767 1362 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.62:35805)
W20250901 01:53:25.849846 1362 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.62:35805)
W20250901 01:53:25.850934 1241 connection.cc:537] server connection from 127.0.0.1:58732 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:53:26.167501 2011 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:26.167740 2011 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:26.168433 2011 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:26.170346 1891 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:26.170732 1894 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:26.170873 1894 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:26.171161 2011 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250901 01:53:26.707715 2022 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:26.707893 2022 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:26.708608 2022 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:26.710732 1891 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:26.711119 1894 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:26.711241 1894 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:26.711498 2022 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250901 01:53:26.903174 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task1fHXBS/build/asan/bin/kudu pid 1866
I20250901 01:53:26.972432 1937 generic_service.cc:196] Checking for leaks (request via RPC)
I20250901 01:53:27.086467 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1fHXBS/build/asan/bin/kudu with pid 1866
I20250901 01:53:28.001747 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1fHXBS/build/asan/bin/kudu
/tmp/dist-test-task1fHXBS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:33025
--webserver_interface=127.0.104.61
--webserver_port=34857
--builtin_ntp_servers=127.0.104.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250901 01:53:28.355840 2044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:53:28.356173 2044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:53:28.356266 2044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 01:53:28.365193 2044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 01:53:28.365309 2044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:53:28.365366 2044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 01:53:28.365415 2044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 01:53:28.376379 2044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=34857
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
I20250901 01:53:28.378087 2044 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:53:28.379980 2044 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 01:53:28.388883 2051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:28.388916 2050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:28.389467 2053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:28.390097 2044 server_base.cc:1047] running on GCE node
I20250901 01:53:28.391115 2044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:53:28.392789 2044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:53:28.393996 2044 hybrid_clock.cc:648] HybridClock initialized: now 1756691608393930 us; error 63 us; skew 500 ppm
I20250901 01:53:28.394380 2044 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:53:28.396873 2044 webserver.cc:480] Webserver started at http://127.0.104.61:34857/ using document root <none> and password file <none>
I20250901 01:53:28.397671 2044 fs_manager.cc:362] Metadata directory not provided
I20250901 01:53:28.397785 2044 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 01:53:28.403954 2044 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20250901 01:53:28.407370 2059 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:28.408967 2044 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250901 01:53:28.409165 2044 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data,/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
uuid: "b557230fc3b142c4bf5f6f58d241905f"
format_stamp: "Formatted at 2025-09-01 01:53:13 on dist-test-slave-r4vn"
I20250901 01:53:28.409956 2044 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:53:28.449976 2044 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 01:53:28.450918 2044 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:53:28.451216 2044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:53:28.472173 2044 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:33025
I20250901 01:53:28.472195 2110 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:33025 every 8 connection(s)
I20250901 01:53:28.474022 2044 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/info.pb
I20250901 01:53:28.481797 2111 sys_catalog.cc:263] Verifying existing consensus state
I20250901 01:53:28.482509 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1fHXBS/build/asan/bin/kudu as pid 2044
I20250901 01:53:28.486568 2111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Bootstrap starting.
I20250901 01:53:28.505679 2111 log.cc:826] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Log is configured to *not* fsync() on all Append() calls
I20250901 01:53:28.580679 2111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Bootstrap replayed 1/1 log segments. Stats: ops{read=97 overwritten=0 applied=97 ignored=0} inserts{seen=64 ignored=0} mutations{seen=76 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 01:53:28.581216 2111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Bootstrap complete.
I20250901 01:53:28.586486 2111 raft_consensus.cc:357] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:28.587268 2111 raft_consensus.cc:738] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: b557230fc3b142c4bf5f6f58d241905f, State: Initialized, Role: FOLLOWER
I20250901 01:53:28.587957 2111 consensus_queue.cc:260] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 97, Last appended: 6.97, Last appended by leader: 97, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:28.588816 2116 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:28.589095 2116 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:28.589957 2111 sys_catalog.cc:564] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: configured and running, proceeding with master startup.
I20250901 01:53:28.601002 2044 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:53:28.606735 2044 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:53:28.608299 2130 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:28.608428 2130 catalog_manager.cc:1554] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: loading cluster ID for follower catalog manager: success
I20250901 01:53:28.618680 1071 heartbeater.cc:344] Connected to a master server at 127.0.104.61:33025
I20250901 01:53:28.618891 2130 catalog_manager.cc:1576] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: acquiring CA information for follower catalog manager: success
I20250901 01:53:28.622579 2076 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" instance_seqno: 1756691595802033) as {username='slave'} at 127.0.104.1:59707; Asking this server to re-register.
I20250901 01:53:28.622938 2130 catalog_manager.cc:1604] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 01:53:28.623718 1071 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:28.626499 2076 ts_manager.cc:194] Registered new tserver with Master: b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:28.678102 1214 heartbeater.cc:344] Connected to a master server at 127.0.104.61:33025
I20250901 01:53:28.680775 2076 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" instance_seqno: 1756691596380664) as {username='slave'} at 127.0.104.2:42941; Asking this server to re-register.
I20250901 01:53:28.681654 1214 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:28.683333 2076 ts_manager.cc:194] Registered new tserver with Master: e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:28.684595 1354 heartbeater.cc:344] Connected to a master server at 127.0.104.61:33025
I20250901 01:53:28.687085 2076 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ad2ba2450480454388723cd729cc438e" instance_seqno: 1756691596941203) as {username='slave'} at 127.0.104.3:36275; Asking this server to re-register.
I20250901 01:53:28.688285 1354 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:28.690220 2076 ts_manager.cc:194] Registered new tserver with Master: ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:29.086288 2116 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:29.086587 2116 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:29.088498 2116 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:29.089274 2060 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:29.090715 2060 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:29.090929 2063 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:29.091028 2063 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:29.091372 2116 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250901 01:53:29.819685 2153 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:29.819914 2153 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:29.820595 2153 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:29.822662 2060 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:29.822853 2063 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:29.823014 2063 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:29.823289 2153 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250901 01:53:30.421288 2167 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:30.421496 2167 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:30.422272 2167 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:30.424078 2060 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:30.424244 2063 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:30.424369 2063 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:30.424613 2167 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250901 01:53:31.076435 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task1fHXBS/build/asan/bin/kudu pid 2044
I20250901 01:53:31.083194 2106 generic_service.cc:196] Checking for leaks (request via RPC)
I20250901 01:53:31.194273 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1fHXBS/build/asan/bin/kudu with pid 2044
W20250901 01:53:31.215214 1242 connection.cc:537] client connection to 127.0.104.61:33025 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:53:32.052085 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1fHXBS/build/asan/bin/kudu
/tmp/dist-test-task1fHXBS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:33025
--webserver_interface=127.0.104.61
--webserver_port=34857
--builtin_ntp_servers=127.0.104.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250901 01:53:32.410348 2192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:53:32.410784 2192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:53:32.410916 2192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 01:53:32.422366 2192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 01:53:32.422472 2192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:53:32.422530 2192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 01:53:32.422580 2192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 01:53:32.433321 2192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=34857
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
I20250901 01:53:32.435031 2192 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:53:32.436780 2192 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 01:53:32.445246 2204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:32.445246 2203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:32.445765 2206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:32.446640 2192 server_base.cc:1047] running on GCE node
I20250901 01:53:32.447510 2192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:53:32.449131 2192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:53:32.450349 2192 hybrid_clock.cc:648] HybridClock initialized: now 1756691612450299 us; error 46 us; skew 500 ppm
I20250901 01:53:32.450708 2192 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:53:32.453233 2192 webserver.cc:480] Webserver started at http://127.0.104.61:34857/ using document root <none> and password file <none>
I20250901 01:53:32.454073 2192 fs_manager.cc:362] Metadata directory not provided
I20250901 01:53:32.454226 2192 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 01:53:32.460140 2192 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20250901 01:53:32.463368 2212 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:32.464864 2192 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250901 01:53:32.465070 2192 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data,/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
uuid: "b557230fc3b142c4bf5f6f58d241905f"
format_stamp: "Formatted at 2025-09-01 01:53:13 on dist-test-slave-r4vn"
I20250901 01:53:32.465806 2192 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:53:32.479558 2192 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 01:53:32.480559 2192 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:53:32.480891 2192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:53:32.503089 2192 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:33025
I20250901 01:53:32.503085 2264 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:33025 every 8 connection(s)
I20250901 01:53:32.504865 2192 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/info.pb
I20250901 01:53:32.512249 2265 sys_catalog.cc:263] Verifying existing consensus state
I20250901 01:53:32.512467 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1fHXBS/build/asan/bin/kudu as pid 2192
I20250901 01:53:32.516861 2265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Bootstrap starting.
I20250901 01:53:32.538414 2265 log.cc:826] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Log is configured to *not* fsync() on all Append() calls
I20250901 01:53:32.612277 2265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Bootstrap replayed 1/1 log segments. Stats: ops{read=97 overwritten=0 applied=97 ignored=0} inserts{seen=64 ignored=0} mutations{seen=76 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 01:53:32.613061 2265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Bootstrap complete.
I20250901 01:53:32.618706 2265 raft_consensus.cc:357] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:32.619438 2265 raft_consensus.cc:738] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: b557230fc3b142c4bf5f6f58d241905f, State: Initialized, Role: FOLLOWER
I20250901 01:53:32.620124 2265 consensus_queue.cc:260] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 97, Last appended: 6.97, Last appended by leader: 97, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:32.620987 2269 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:32.621261 2269 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:32.622105 2265 sys_catalog.cc:564] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: configured and running, proceeding with master startup.
I20250901 01:53:32.631994 2192 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:53:32.637852 2192 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:53:32.640985 2283 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:32.641119 2283 catalog_manager.cc:1554] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: loading cluster ID for follower catalog manager: success
I20250901 01:53:32.648762 2283 catalog_manager.cc:1576] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: acquiring CA information for follower catalog manager: success
I20250901 01:53:32.653199 2283 catalog_manager.cc:1604] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 01:53:32.727639 1354 heartbeater.cc:344] Connected to a master server at 127.0.104.61:33025
I20250901 01:53:32.727720 1071 heartbeater.cc:344] Connected to a master server at 127.0.104.61:33025
I20250901 01:53:32.730876 2229 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" instance_seqno: 1756691595802033) as {username='slave'} at 127.0.104.1:38191; Asking this server to re-register.
I20250901 01:53:32.730921 2228 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ad2ba2450480454388723cd729cc438e" instance_seqno: 1756691596941203) as {username='slave'} at 127.0.104.3:57305; Asking this server to re-register.
I20250901 01:53:32.732820 1071 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:32.732978 1354 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:32.735318 2228 ts_manager.cc:194] Registered new tserver with Master: ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:32.735464 2229 ts_manager.cc:194] Registered new tserver with Master: b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:32.925490 1214 heartbeater.cc:344] Connected to a master server at 127.0.104.61:33025
I20250901 01:53:32.927973 2228 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" instance_seqno: 1756691596380664) as {username='slave'} at 127.0.104.2:47879; Asking this server to re-register.
I20250901 01:53:32.928642 1214 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:32.930302 2228 ts_manager.cc:194] Registered new tserver with Master: e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:33.164716 2294 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:33.165045 2294 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:33.166992 2294 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:33.167950 2213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:33.169251 2216 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:53:33.169461 2213 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:53:33.169605 2213 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:33.169971 2294 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250901 01:53:33.687635 2301 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:33.687844 2301 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:33.688812 2301 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:33.690567 2213 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:33.690742 2216 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:33.690909 2216 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:33.691152 2301 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250901 01:53:34.134842 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 6: this message will repeat every 5th retry.
I20250901 01:53:34.331895 2315 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:34.332088 2315 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:34.332785 2315 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:34.335412 2213 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:34.335677 2216 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:34.335811 2216 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:34.336050 2315 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250901 01:53:34.442395 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 6: this message will repeat every 5th retry.
I20250901 01:53:35.925097 2336 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:35.925336 2336 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:35.926021 2336 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:35.928536 2213 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:35.928678 2216 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:35.928825 2216 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:35.929085 2336 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250901 01:53:36.021803 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1fHXBS/build/asan/bin/kudu
/tmp/dist-test-task1fHXBS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35805
--webserver_interface=127.0.104.62
--webserver_port=32963
--builtin_ntp_servers=127.0.104.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250901 01:53:36.373849 2344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:53:36.374265 2344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:53:36.374370 2344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 01:53:36.383147 2344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 01:53:36.383267 2344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:53:36.383325 2344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 01:53:36.383365 2344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 01:53:36.395603 2344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=32963
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
I20250901 01:53:36.397389 2344 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:53:36.399745 2344 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 01:53:36.408414 2351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:36.408403 2352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:36.409535 2354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:36.410601 2344 server_base.cc:1047] running on GCE node
I20250901 01:53:36.411504 2344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:53:36.413168 2344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:53:36.414372 2344 hybrid_clock.cc:648] HybridClock initialized: now 1756691616414325 us; error 53 us; skew 500 ppm
I20250901 01:53:36.414726 2344 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:53:36.416968 2344 webserver.cc:480] Webserver started at http://127.0.104.62:32963/ using document root <none> and password file <none>
I20250901 01:53:36.417752 2344 fs_manager.cc:362] Metadata directory not provided
I20250901 01:53:36.417884 2344 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 01:53:36.423635 2344 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20250901 01:53:36.426760 2360 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:36.428480 2344 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250901 01:53:36.428815 2344 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data,/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21"
format_stamp: "Formatted at 2025-09-01 01:53:13 on dist-test-slave-r4vn"
I20250901 01:53:36.429792 2344 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:53:36.444996 2344 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 01:53:36.445953 2344 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:53:36.446307 2344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:53:36.466414 2344 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35805
I20250901 01:53:36.466418 2411 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35805 every 8 connection(s)
I20250901 01:53:36.468355 2344 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/info.pb
I20250901 01:53:36.472785 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1fHXBS/build/asan/bin/kudu as pid 2344
I20250901 01:53:36.476506 2412 sys_catalog.cc:263] Verifying existing consensus state
I20250901 01:53:36.481736 2412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Bootstrap starting.
I20250901 01:53:36.500725 2412 log.cc:826] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Log is configured to *not* fsync() on all Append() calls
I20250901 01:53:36.514853 1212 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35805
I20250901 01:53:36.517983 1069 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35805
I20250901 01:53:36.582039 2412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=97 overwritten=0 applied=96 ignored=0} inserts{seen=64 ignored=0} mutations{seen=72 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250901 01:53:36.582705 2412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Bootstrap complete.
I20250901 01:53:36.588021 2412 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:36.590742 2412 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c35f5a36eeb4a93aecdd71e2d1c3a21, State: Initialized, Role: FOLLOWER
I20250901 01:53:36.591522 2412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 96, Last appended: 6.97, Last appended by leader: 97, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:36.592356 2423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:36.592641 2423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:36.593525 2412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 01:53:36.611756 2437 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:36.611961 2437 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: loading cluster ID for follower catalog manager: success
I20250901 01:53:36.617800 2437 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: acquiring CA information for follower catalog manager: success
I20250901 01:53:36.622005 2437 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 01:53:36.628988 2344 master_runner.cc:186] Error getting master registration for 127.0.104.60:36985: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111),
I20250901 01:53:36.632716 2344 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:53:36.658968 2377 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" instance_seqno: 1756691596380664) as {username='slave'} at 127.0.104.2:54247; Asking this server to re-register.
I20250901 01:53:36.659745 1212 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:36.661787 2377 ts_manager.cc:194] Registered new tserver with Master: e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:36.705683 1352 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35805
I20250901 01:53:36.707688 2377 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ad2ba2450480454388723cd729cc438e" instance_seqno: 1756691596941203) as {username='slave'} at 127.0.104.3:35037; Asking this server to re-register.
I20250901 01:53:36.708323 1352 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:36.709905 2377 ts_manager.cc:194] Registered new tserver with Master: ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:37.022439 2377 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" instance_seqno: 1756691595802033) as {username='slave'} at 127.0.104.1:55551; Asking this server to re-register.
I20250901 01:53:37.023283 1069 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:37.024901 2377 ts_manager.cc:194] Registered new tserver with Master: b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:37.046535 2423 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:37.046845 2423 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:37.048686 2423 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:37.049329 2364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:37.049322 2240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 7 candidate_status { last_received { term: 6 index: 97 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f" is_pre_election: true
I20250901 01:53:37.049774 2240 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c35f5a36eeb4a93aecdd71e2d1c3a21 in term 6.
I20250901 01:53:37.050450 2363 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, b557230fc3b142c4bf5f6f58d241905f; no voters:
I20250901 01:53:37.050843 2423 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250901 01:53:37.050916 2364 leader_election.cc:336] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:37.050980 2423 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:37.051095 2423 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 6 FOLLOWER]: Advancing to term 7
I20250901 01:53:37.054495 2423 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:37.055188 2423 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 7 election: Requested vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:37.055743 2240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 7 candidate_status { last_received { term: 6 index: 97 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f"
I20250901 01:53:37.055972 2240 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 6 FOLLOWER]: Advancing to term 7
W20250901 01:53:37.056646 2364 leader_election.cc:336] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:37.059854 2240 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c35f5a36eeb4a93aecdd71e2d1c3a21 in term 7.
I20250901 01:53:37.060317 2363 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, b557230fc3b142c4bf5f6f58d241905f; no voters: 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:37.060668 2423 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 7 FOLLOWER]: Leader election won for term 7
I20250901 01:53:37.061056 2423 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 7 LEADER]: Becoming Leader. State: Replica: 2c35f5a36eeb4a93aecdd71e2d1c3a21, State: Running, Role: LEADER
I20250901 01:53:37.061515 2423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 6.97, Last appended by leader: 97, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:37.064384 2445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c35f5a36eeb4a93aecdd71e2d1c3a21. Latest consensus state: current_term: 7 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:37.064673 2445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: LEADER
W20250901 01:53:37.511559 2364 consensus_peers.cc:489] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 -> Peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Couldn't send request to peer 835a442fbc4c48eb9148ca677e410a59. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:53:37.584995 2240 raft_consensus.cc:1273] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 7 FOLLOWER]: Refusing update from remote peer 2c35f5a36eeb4a93aecdd71e2d1c3a21: Log matching property violated. Preceding OpId in replica: term: 6 index: 97. Preceding OpId from leader: term: 7 index: 98. (index mismatch)
I20250901 01:53:37.586006 2445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20250901 01:53:37.589838 2453 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c35f5a36eeb4a93aecdd71e2d1c3a21. Latest consensus state: current_term: 7 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:37.590029 2453 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:37.592074 2455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:37.592653 2455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:37.592905 2453 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:37.593114 2453 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:37.655189 2192 master_runner.cc:186] Error getting master registration for 127.0.104.60:36985: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111),
I20250901 01:53:37.655398 2192 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:53:37.710457 2447 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 01:53:37.713512 2447 catalog_manager.cc:671] Loaded metadata for table default.table_bb66c7b604104254a48d435595b83fc4 [id=267f90dc52214e7986546d0e99a7f23a]
I20250901 01:53:37.714089 2447 catalog_manager.cc:671] Loaded metadata for table default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19]
I20250901 01:53:37.714602 2447 catalog_manager.cc:671] Loaded metadata for table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]
I20250901 01:53:37.715070 2447 catalog_manager.cc:671] Loaded metadata for table default.table_8082166db25745e78737b640063b3219 [id=37fc4ed840564a8780eeb9847b373712]
I20250901 01:53:37.715533 2447 catalog_manager.cc:671] Loaded metadata for table default.table_8f21a26e011c4ed1b1ae7cb364e09195 [id=39123cfe4f62420681f8d679d2d376b8]
I20250901 01:53:37.715914 2447 catalog_manager.cc:671] Loaded metadata for table default.table_9f3451ed25714464ad16b3e94214d0e3 [id=5575a859d470414e9a1bf99d5738fc3a]
I20250901 01:53:37.716382 2447 catalog_manager.cc:671] Loaded metadata for table default.table_da487bf0d57f439bb9da2ec57880a73b [id=7185f28e50df4fb9a4bf41cc5b73169d]
I20250901 01:53:37.716706 2447 catalog_manager.cc:671] Loaded metadata for table default.table_167322850c734684b896b875526a5c6e [id=7a69995eebe446629422a7fe0924e58a]
I20250901 01:53:37.717018 2447 catalog_manager.cc:671] Loaded metadata for table default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d]
I20250901 01:53:37.717342 2447 catalog_manager.cc:671] Loaded metadata for table default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98]
I20250901 01:53:37.717649 2447 catalog_manager.cc:671] Loaded metadata for table default.table_1ff7214e853748e58232b9bea0cb6912 [id=af5c975082de48cfa7295b2d8bd16635]
I20250901 01:53:37.717967 2447 catalog_manager.cc:671] Loaded metadata for table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647]
I20250901 01:53:37.718336 2447 catalog_manager.cc:671] Loaded metadata for table default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9]
I20250901 01:53:37.718667 2447 catalog_manager.cc:671] Loaded metadata for table default.table_facefcd0049d44a2b465d8315527aa08 [id=d0ce834bce8e4a3fa795fb9e3b6b06a7]
I20250901 01:53:37.718976 2447 catalog_manager.cc:671] Loaded metadata for table default.table_c95fbf8fa7a64aa18002f1e268cdea68 [id=d830537abb3c488ca7b925a8a491432a]
I20250901 01:53:37.719440 2447 catalog_manager.cc:671] Loaded metadata for table default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5]
I20250901 01:53:37.719764 2447 catalog_manager.cc:671] Loaded metadata for table default.table_99b272393a1b40b4ae2d3bbaedc9ea60 [id=f868207bffb24b6795d0f636974d231d]
I20250901 01:53:37.721876 2447 tablet_loader.cc:96] loaded metadata for tablet 089fe953bba0403db288afefaf750c10 (table default.table_99b272393a1b40b4ae2d3bbaedc9ea60 [id=f868207bffb24b6795d0f636974d231d])
I20250901 01:53:37.722276 2447 tablet_loader.cc:96] loaded metadata for tablet 0e88f2701be74589b1337bd6f4ef87db (table default.table_da487bf0d57f439bb9da2ec57880a73b [id=7185f28e50df4fb9a4bf41cc5b73169d])
I20250901 01:53:37.722760 2447 tablet_loader.cc:96] loaded metadata for tablet 3733efc4223d49a28f69cb598bdf7c94 (table default.table_facefcd0049d44a2b465d8315527aa08 [id=d0ce834bce8e4a3fa795fb9e3b6b06a7])
I20250901 01:53:37.723436 2447 tablet_loader.cc:96] loaded metadata for tablet 5274573577374b47b99fa310a5ca83bb (table default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19])
I20250901 01:53:37.723721 2447 tablet_loader.cc:96] loaded metadata for tablet 586fbc1ef054435aaaea9ae961f15115 (table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647])
I20250901 01:53:37.724262 2447 tablet_loader.cc:96] loaded metadata for tablet 80d0e50666a34df39a4cc09977effe5c (table default.table_167322850c734684b896b875526a5c6e [id=7a69995eebe446629422a7fe0924e58a])
I20250901 01:53:37.724777 2447 tablet_loader.cc:96] loaded metadata for tablet 8dd9f23bffa543a490c018108ad41051 (table default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98])
I20250901 01:53:37.725059 2447 tablet_loader.cc:96] loaded metadata for tablet 8f913036f34345fb8b20b3c9f431979e (table default.table_8f21a26e011c4ed1b1ae7cb364e09195 [id=39123cfe4f62420681f8d679d2d376b8])
I20250901 01:53:37.725840 2447 tablet_loader.cc:96] loaded metadata for tablet a6af5a07f3ca4fe4a8808b913551c091 (table default.table_1ff7214e853748e58232b9bea0cb6912 [id=af5c975082de48cfa7295b2d8bd16635])
I20250901 01:53:37.726310 2447 tablet_loader.cc:96] loaded metadata for tablet ad4ea5cf599f4719960fdf0b2e6fc054 (table default.table_9f3451ed25714464ad16b3e94214d0e3 [id=5575a859d470414e9a1bf99d5738fc3a])
I20250901 01:53:37.726629 2447 tablet_loader.cc:96] loaded metadata for tablet b0064e7090a54fb18ee9a761ddec2c83 (table default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9])
I20250901 01:53:37.726902 2447 tablet_loader.cc:96] loaded metadata for tablet b302bfaed5104909a1d18f7529398348 (table default.table_8082166db25745e78737b640063b3219 [id=37fc4ed840564a8780eeb9847b373712])
I20250901 01:53:37.727893 2447 tablet_loader.cc:96] loaded metadata for tablet d8709bb983864d9f9e65eaf8f21a13f8 (table default.table_c95fbf8fa7a64aa18002f1e268cdea68 [id=d830537abb3c488ca7b925a8a491432a])
I20250901 01:53:37.728159 2447 tablet_loader.cc:96] loaded metadata for tablet de13eb65b0df4b8ebc049ff753123d53 (table default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5])
I20250901 01:53:37.728449 2447 tablet_loader.cc:96] loaded metadata for tablet debd885f996642faa60cfa5af0283b5f (table default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d])
I20250901 01:53:37.728711 2447 tablet_loader.cc:96] loaded metadata for tablet e4df70f8191e4115991d42178cb67bd7 (table default.table_bb66c7b604104254a48d435595b83fc4 [id=267f90dc52214e7986546d0e99a7f23a])
I20250901 01:53:37.728982 2447 tablet_loader.cc:96] loaded metadata for tablet f81df1b890684c9cb01597c5bfebcefa (table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd])
I20250901 01:53:37.729228 2447 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 01:53:37.730614 2447 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:37.730698 2447 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 01:53:37.732242 2447 catalog_manager.cc:1506] Loading token signing keys...
I20250901 01:53:37.733453 2447 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Loaded TSK: 0
I20250901 01:53:37.734404 2447 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 01:53:37.735630 2447 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 01:53:37.736908 2447 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 22
I20250901 01:53:37.743587 2377 master_service.cc:948] ReplaceTablet: received request to replace tablet f81df1b890684c9cb01597c5bfebcefa from {username='slave'} at 127.0.0.1:41614
I20250901 01:53:37.744288 2375 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "default.Table_c7e1601c717640bba2d6ca6aa5478dbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:37.744820 2374 master_service.cc:948] ReplaceTablet: received request to replace tablet b56b8dbd4e5b4892aa3b50409fc4584c from {username='slave'} at 127.0.0.1:41614
I20250901 01:53:37.745827 2376 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "default.Table_820e14b2e7cf4bd7ad59ea83d8cb4d60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:37.746827 2374 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "default.Table_ed3c253535084a3399b30083bb388f72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:37.747639 2371 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "default.Table_2368f4dfae714931b490e2544bb27666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:37.747779 2370 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "default.Table_c1e876d5919f4c1f98a0ef9d1fcb9edc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:37.748181 2373 master_service.cc:948] ReplaceTablet: received request to replace tablet a774cb6ccef24d958ba6c898c5e4786c from {username='slave'} at 127.0.0.1:41614
I20250901 01:53:37.748893 2373 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "default.Table_56fde480a00a4e62b84eedba22b0bade"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:37.749329 2372 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "default.Table_facefcd0049d44a2b465d8315527aa08" } modify_external_catalogs: true
I20250901 01:53:37.749588 2372 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "default.Table_facefcd0049d44a2b465d8315527aa08" } modify_external_catalogs: true
I20250901 01:53:37.749891 2368 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "default.Table_cbe4bb8385f54f7f9359f495d8c663a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:37.750424 2369 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "default.Table_f6c5cfd184ce474b883ce77bd225eb9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:37.756971 2377 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Sending DeleteTablet for 3 replicas of tablet f81df1b890684c9cb01597c5bfebcefa
I20250901 01:53:37.759280 2377 catalog_manager.cc:6652] ReplaceTablet: tablet f81df1b890684c9cb01597c5bfebcefa deleted and replaced by tablet 83b971ea891840229df1f7ed3e0d1d92
F20250901 01:53:37.759913 2436 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 5 out of 13 received events: NotificationEvent(eventId=23, eventTime=1756691601, eventType=CREATE_TABLE, dbName=default, tableName=table_98a9d1400149426bab73e31aea263f5b, message={"server":"","servicePrincipal":"","db":"default","table":"table_98a9d1400149426bab73e31aea263f5b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_98a9d1400149426bab73e31aea263f5b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691601},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_98a9d1400149426bab73e31aea263f5b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_98a9d1400149426bab73e31aea263f5b\",\"kudu.table_id\":\"49527ae170f04658b8b1eb01b07ffd04\",\"transient_lastDdlTime\":\"1756691601\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1756691601, eventType=ALTER_TABLE, dbName=default, tableName=table_757f6c2777014fe1b0686fc1405b4b5f, message={"server":"","servicePrincipal":"","db":"default","table":"table_61dfd07a8f784854b5d9e104e010daaa","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_61dfd07a8f784854b5d9e104e010daaa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691599},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_61dfd07a8f784854b5d9e104e010daaa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_61dfd07a8f784854b5d9e104e010daaa\",\"kudu.table_id\":\"ae0da2fb2d1542308ccb9a95247c1d98\",\"transient_lastDdlTime\":\"1756691599\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_757f6c2777014fe1b0686fc1405b4b5f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691599},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_757f6c2777014fe1b0686fc1405b4b5f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_757f6c2777014fe1b0686fc1405b4b5f\",\"kudu.table_id\":\"ae0da2fb2d1542308ccb9a95247c1d98\",\"transient_lastDdlTime\":\"1756691599\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_c2864548fe1a4719b73cf55d0933e538, message={"server":"","servicePrincipal":"","db":"default","table":"table_c2864548fe1a4719b73cf55d0933e538","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c2864548fe1a4719b73cf55d0933e538\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691598},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_c2864548fe1a4719b73cf55d0933e538\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c2864548fe1a4719b73cf55d0933e538\",\"kudu.table_id\":\"31d721fb8b204568a0190dda6def01cd\",\"transient_lastDdlTime\":\"1756691598\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_56fde480a00a4e62b84eedba22b0bade, message={"server":"","servicePrincipal":"","db":"default","table":"table_56fde480a00a4e62b84eedba22b0bade","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_56fde480a00a4e62b84eedba22b0bade\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691601},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_56fde480a00a4e62b84eedba22b0bade\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_56fde480a00a4e62b84eedba22b0bade\",\"kudu.table_id\":\"181eeac3e2394bcbaf3071e3fad223d8\",\"transient_lastDdlTime\":\"1756691601\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_7722e49197be4ca9b9c7e67052f07c4b, message={"server":"","servicePrincipal":"","db":"default","table":"table_7722e49197be4ca9b9c7e67052f07c4b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_7722e49197be4ca9b9c7e67052f07c4b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691598},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_7722e49197be4ca9b9c7e67052f07c4b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7722e49197be4ca9b9c7e67052f07c4b\",\"kudu.table_id\":\"2e950743029d48a49f05207cd5faee19\",\"transient_lastDdlTime\":\"1756691598\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250901 01:53:37.759806 2377 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "default.Table_98a9d1400149426bab73e31aea263f5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:37.786375 1286 tablet_service.cc:1515] Processing DeleteTablet for tablet f81df1b890684c9cb01597c5bfebcefa with delete_type TABLET_DATA_DELETED (replaced by tablet 83b971ea891840229df1f7ed3e0d1d92) from {username='slave'} at 127.0.0.1:37414
I20250901 01:53:37.789445 2471 tablet_replica.cc:331] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e: stopping tablet replica
I20250901 01:53:37.789697 2471 raft_consensus.cc:2241] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:37.789954 2471 raft_consensus.cc:2270] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:37.790001 1004 tablet_service.cc:1515] Processing DeleteTablet for tablet f81df1b890684c9cb01597c5bfebcefa with delete_type TABLET_DATA_DELETED (replaced by tablet 83b971ea891840229df1f7ed3e0d1d92) from {username='slave'} at 127.0.0.1:60524
I20250901 01:53:37.790671 2471 ts_tablet_manager.cc:1905] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:37.790735 2472 tablet_replica.cc:331] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853: stopping tablet replica
I20250901 01:53:37.791013 2472 raft_consensus.cc:2241] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:53:37.791452 2472 raft_consensus.cc:2270] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:37.791586 1146 tablet_service.cc:1515] Processing DeleteTablet for tablet f81df1b890684c9cb01597c5bfebcefa with delete_type TABLET_DATA_DELETED (replaced by tablet 83b971ea891840229df1f7ed3e0d1d92) from {username='slave'} at 127.0.0.1:43464
I20250901 01:53:37.792091 2472 ts_tablet_manager.cc:1905] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:37.792300 2473 tablet_replica.cc:331] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5: stopping tablet replica
I20250901 01:53:37.792532 2473 raft_consensus.cc:2241] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:53:37.792766 2473 raft_consensus.cc:2270] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:53:37.793316 2473 ts_tablet_manager.cc:1905] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:53:37.795704 2471 ts_tablet_manager.cc:1918] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:37.795949 2471 log.cc:1199] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-2/wal/wals/f81df1b890684c9cb01597c5bfebcefa
I20250901 01:53:37.796331 2471 ts_tablet_manager.cc:1939] T f81df1b890684c9cb01597c5bfebcefa P ad2ba2450480454388723cd729cc438e: Deleting consensus metadata
I20250901 01:53:37.798467 2362 catalog_manager.cc:4928] TS ad2ba2450480454388723cd729cc438e (127.0.104.3:42927): tablet f81df1b890684c9cb01597c5bfebcefa (table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]) successfully deleted
@ 0x7f7a01163dcd google::LogMessage::Fail() at ??:0
I20250901 01:53:37.799054 2472 ts_tablet_manager.cc:1918] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:37.799173 2472 log.cc:1199] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-0/wal/wals/f81df1b890684c9cb01597c5bfebcefa
I20250901 01:53:37.799695 2472 ts_tablet_manager.cc:1939] T f81df1b890684c9cb01597c5bfebcefa P b65b3154aefd49f7a37f70e9207e9853: Deleting consensus metadata
I20250901 01:53:37.799754 2473 ts_tablet_manager.cc:1918] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:53:37.799876 2473 log.cc:1199] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5: Deleting WAL directory at /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/ts-1/wal/wals/f81df1b890684c9cb01597c5bfebcefa
@ 0x7f7a01167b93 google::LogMessage::SendToLog() at ??:0
I20250901 01:53:37.800282 2473 ts_tablet_manager.cc:1939] T f81df1b890684c9cb01597c5bfebcefa P e3c774a4edf44d6098a2332e734967a5: Deleting consensus metadata
I20250901 01:53:37.801115 2364 catalog_manager.cc:4928] TS b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): tablet f81df1b890684c9cb01597c5bfebcefa (table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]) successfully deleted
@ 0x7f7a011637cc google::LogMessage::Flush() at ??:0
I20250901 01:53:37.801287 2361 catalog_manager.cc:4928] TS e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245): tablet f81df1b890684c9cb01597c5bfebcefa (table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]) successfully deleted
@ 0x7f7a01163a11 google::LogMessage::~LogMessage() at ??:0
I20250901 01:53:37.803764 1004 tablet_service.cc:1468] Processing CreateTablet for tablet 83b971ea891840229df1f7ed3e0d1d92 (DEFAULT_TABLE table=default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:37.804199 1004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83b971ea891840229df1f7ed3e0d1d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:37.804181 1286 tablet_service.cc:1468] Processing CreateTablet for tablet 83b971ea891840229df1f7ed3e0d1d92 (DEFAULT_TABLE table=default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:37.804237 1146 tablet_service.cc:1468] Processing CreateTablet for tablet 83b971ea891840229df1f7ed3e0d1d92 (DEFAULT_TABLE table=default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:53:37.804576 1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83b971ea891840229df1f7ed3e0d1d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:53:37.804730 1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83b971ea891840229df1f7ed3e0d1d92. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f7a17598794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250901 01:53:37.819453 2476 tablet_bootstrap.cc:492] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853: Bootstrap starting.
I20250901 01:53:37.821213 2475 tablet_bootstrap.cc:492] T 83b971ea891840229df1f7ed3e0d1d92 P ad2ba2450480454388723cd729cc438e: Bootstrap starting.
I20250901 01:53:37.823195 2475 tablet_bootstrap.cc:654] T 83b971ea891840229df1f7ed3e0d1d92 P ad2ba2450480454388723cd729cc438e: Neither blocks nor log segments found. Creating new log.
@ 0x7f7a17597041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250901 01:53:37.824708 2476 tablet_bootstrap.cc:654] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853: Neither blocks nor log segments found. Creating new log.
I20250901 01:53:37.828301 2477 tablet_bootstrap.cc:492] T 83b971ea891840229df1f7ed3e0d1d92 P e3c774a4edf44d6098a2332e734967a5: Bootstrap starting.
I20250901 01:53:37.830744 2477 tablet_bootstrap.cc:654] T 83b971ea891840229df1f7ed3e0d1d92 P e3c774a4edf44d6098a2332e734967a5: Neither blocks nor log segments found. Creating new log.
@ 0x7f7a1759ecbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f7a1759eb39 std::_Function_handler<>::_M_invoke() at ??:0
I20250901 01:53:37.837163 2476 tablet_bootstrap.cc:492] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853: No bootstrap required, opened a new log
I20250901 01:53:37.837329 2476 ts_tablet_manager.cc:1397] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
I20250901 01:53:37.838135 2476 raft_consensus.cc:357] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:37.838681 2476 raft_consensus.cc:383] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:37.838768 2476 raft_consensus.cc:738] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Initialized, Role: FOLLOWER
@ 0xa774dd std::function<>::operator()() at ??:?
I20250901 01:53:37.839010 2476 consensus_queue.cc:260] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:37.844187 2476 ts_tablet_manager.cc:1428] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
@ 0x7f7a03ec0cc9 kudu::Thread::SuperviseThread() at ??:0
I20250901 01:53:37.845310 2475 tablet_bootstrap.cc:492] T 83b971ea891840229df1f7ed3e0d1d92 P ad2ba2450480454388723cd729cc438e: No bootstrap required, opened a new log
I20250901 01:53:37.845460 2475 ts_tablet_manager.cc:1397] T 83b971ea891840229df1f7ed3e0d1d92 P ad2ba2450480454388723cd729cc438e: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.008s
I20250901 01:53:37.846261 2475 raft_consensus.cc:357] T 83b971ea891840229df1f7ed3e0d1d92 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:37.846446 2475 raft_consensus.cc:383] T 83b971ea891840229df1f7ed3e0d1d92 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:37.846518 2475 raft_consensus.cc:738] T 83b971ea891840229df1f7ed3e0d1d92 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ba2450480454388723cd729cc438e, State: Initialized, Role: FOLLOWER
I20250901 01:53:37.846742 2475 consensus_queue.cc:260] T 83b971ea891840229df1f7ed3e0d1d92 P ad2ba2450480454388723cd729cc438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
@ 0x7f7a072e26db start_thread at ??:0
@ 0x7f7a0001471f clone at ??:0
I20250901 01:53:37.850026 2475 ts_tablet_manager.cc:1428] T 83b971ea891840229df1f7ed3e0d1d92 P ad2ba2450480454388723cd729cc438e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250901 01:53:37.855218 2477 tablet_bootstrap.cc:492] T 83b971ea891840229df1f7ed3e0d1d92 P e3c774a4edf44d6098a2332e734967a5: No bootstrap required, opened a new log
I20250901 01:53:37.855374 2477 ts_tablet_manager.cc:1397] T 83b971ea891840229df1f7ed3e0d1d92 P e3c774a4edf44d6098a2332e734967a5: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.009s
I20250901 01:53:37.856211 2477 raft_consensus.cc:357] T 83b971ea891840229df1f7ed3e0d1d92 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:37.856359 2477 raft_consensus.cc:383] T 83b971ea891840229df1f7ed3e0d1d92 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:53:37.856416 2477 raft_consensus.cc:738] T 83b971ea891840229df1f7ed3e0d1d92 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c774a4edf44d6098a2332e734967a5, State: Initialized, Role: FOLLOWER
I20250901 01:53:37.856598 2477 consensus_queue.cc:260] T 83b971ea891840229df1f7ed3e0d1d92 P e3c774a4edf44d6098a2332e734967a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:37.857066 2477 ts_tablet_manager.cc:1428] T 83b971ea891840229df1f7ed3e0d1d92 P e3c774a4edf44d6098a2332e734967a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250901 01:53:38.044890 1100 connection.cc:537] client connection to 127.0.104.62:35805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:38.045276 1362 connection.cc:537] client connection to 127.0.104.62:35805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:38.046406 1362 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.62:35805)
W20250901 01:53:38.046808 1362 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.62:35805)
W20250901 01:53:38.046980 1362 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.62:35805)
W20250901 01:53:38.047837 1362 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.62:35805)
I20250901 01:53:38.277413 2492 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:38.277639 2492 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:38.278494 2492 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:38.279316 2213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20250901 01:53:38.280644 2216 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:53:38.280831 2213 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:53:38.280977 2213 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:38.281237 2492 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250901 01:53:38.508678 2019 raft_consensus.cc:491] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:38.508895 2019 raft_consensus.cc:513] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:38.509764 2019 leader_election.cc:290] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:38.510360 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83b971ea891840229df1f7ed3e0d1d92" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5" is_pre_election: true
I20250901 01:53:38.510614 1167 raft_consensus.cc:2466] T 83b971ea891840229df1f7ed3e0d1d92 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:38.510576 1306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83b971ea891840229df1f7ed3e0d1d92" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e" is_pre_election: true
I20250901 01:53:38.510797 1306 raft_consensus.cc:2466] T 83b971ea891840229df1f7ed3e0d1d92 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 0.
I20250901 01:53:38.511162 953 leader_election.cc:304] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65b3154aefd49f7a37f70e9207e9853, e3c774a4edf44d6098a2332e734967a5; no voters:
I20250901 01:53:38.511476 2019 raft_consensus.cc:2802] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:53:38.511615 2019 raft_consensus.cc:491] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:38.511715 2019 raft_consensus.cc:3058] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:38.513652 2019 raft_consensus.cc:513] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:38.514278 2019 leader_election.cc:290] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Requested vote from peers e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245), ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:38.514768 1167 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83b971ea891840229df1f7ed3e0d1d92" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c774a4edf44d6098a2332e734967a5"
I20250901 01:53:38.514803 1306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83b971ea891840229df1f7ed3e0d1d92" candidate_uuid: "b65b3154aefd49f7a37f70e9207e9853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2ba2450480454388723cd729cc438e"
I20250901 01:53:38.514986 1306 raft_consensus.cc:3058] T 83b971ea891840229df1f7ed3e0d1d92 P ad2ba2450480454388723cd729cc438e [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:38.515003 1167 raft_consensus.cc:3058] T 83b971ea891840229df1f7ed3e0d1d92 P e3c774a4edf44d6098a2332e734967a5 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:53:38.517823 1306 raft_consensus.cc:2466] T 83b971ea891840229df1f7ed3e0d1d92 P ad2ba2450480454388723cd729cc438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:38.517805 1167 raft_consensus.cc:2466] T 83b971ea891840229df1f7ed3e0d1d92 P e3c774a4edf44d6098a2332e734967a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65b3154aefd49f7a37f70e9207e9853 in term 1.
I20250901 01:53:38.518378 954 leader_election.cc:304] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2ba2450480454388723cd729cc438e, b65b3154aefd49f7a37f70e9207e9853; no voters:
I20250901 01:53:38.518738 2019 raft_consensus.cc:2802] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:53:38.519104 2019 raft_consensus.cc:695] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [term 1 LEADER]: Becoming Leader. State: Replica: b65b3154aefd49f7a37f70e9207e9853, State: Running, Role: LEADER
I20250901 01:53:38.519457 2019 consensus_queue.cc:237] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45315 } } peers { permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 } } peers { permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 } }
I20250901 01:53:39.031247 2504 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:39.031430 2504 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:39.032006 2504 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:39.034016 2213 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:39.034225 2216 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:39.034344 2216 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:39.034564 2504 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250901 01:53:39.583824 2019 consensus_queue.cc:1035] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2ba2450480454388723cd729cc438e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:39.591573 2518 consensus_queue.cc:1035] T 83b971ea891840229df1f7ed3e0d1d92 P b65b3154aefd49f7a37f70e9207e9853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:53:39.697062 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1fHXBS/build/asan/bin/kudu
/tmp/dist-test-task1fHXBS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35805
--webserver_interface=127.0.104.62
--webserver_port=32963
--builtin_ntp_servers=127.0.104.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250901 01:53:39.726897 2525 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:39.727097 2525 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:39.727973 2525 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:39.730000 2213 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:39.730486 2216 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:39.730590 2216 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:39.730871 2525 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250901 01:53:40.056564 2524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:53:40.056910 2524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:53:40.057045 2524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 01:53:40.066030 2524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 01:53:40.066197 2524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:53:40.066298 2524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 01:53:40.066373 2524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 01:53:40.076638 2524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=32963
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
I20250901 01:53:40.078426 2524 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:53:40.080487 2524 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 01:53:40.089373 2534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:40.089349 2533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:40.091179 2536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:40.091727 2524 server_base.cc:1047] running on GCE node
I20250901 01:53:40.093461 2524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:53:40.095520 2524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:53:40.098955 2524 hybrid_clock.cc:648] HybridClock initialized: now 1756691620098896 us; error 43 us; skew 500 ppm
I20250901 01:53:40.099385 2524 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:53:40.102079 2524 webserver.cc:480] Webserver started at http://127.0.104.62:32963/ using document root <none> and password file <none>
I20250901 01:53:40.103174 2524 fs_manager.cc:362] Metadata directory not provided
I20250901 01:53:40.103353 2524 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 01:53:40.109544 2524 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250901 01:53:40.113034 2546 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:40.114724 2524 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250901 01:53:40.114919 2524 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data,/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21"
format_stamp: "Formatted at 2025-09-01 01:53:13 on dist-test-slave-r4vn"
I20250901 01:53:40.115694 2524 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:53:40.143872 2524 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 01:53:40.144976 2524 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:53:40.145310 2524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:53:40.165913 2524 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35805
I20250901 01:53:40.165931 2597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35805 every 8 connection(s)
I20250901 01:53:40.167712 2524 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/info.pb
I20250901 01:53:40.168763 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1fHXBS/build/asan/bin/kudu as pid 2524
I20250901 01:53:40.176573 2598 sys_catalog.cc:263] Verifying existing consensus state
I20250901 01:53:40.181406 2598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Bootstrap starting.
I20250901 01:53:40.202420 2598 log.cc:826] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Log is configured to *not* fsync() on all Append() calls
I20250901 01:53:40.275759 2598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=100 overwritten=0 applied=100 ignored=0} inserts{seen=65 ignored=0} mutations{seen=78 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 01:53:40.276366 2598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Bootstrap complete.
I20250901 01:53:40.281719 2598 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:40.282509 2598 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c35f5a36eeb4a93aecdd71e2d1c3a21, State: Initialized, Role: FOLLOWER
I20250901 01:53:40.283237 2598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 100, Last appended: 7.100, Last appended by leader: 100, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:40.284243 2602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:40.284600 2602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:40.285396 2598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 01:53:40.305992 2616 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:40.306201 2616 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: loading cluster ID for follower catalog manager: success
I20250901 01:53:40.311584 2616 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: acquiring CA information for follower catalog manager: success
I20250901 01:53:40.315603 2616 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 01:53:40.333025 2524 master_runner.cc:186] Error getting master registration for 127.0.104.60:36985: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111),
I20250901 01:53:40.336771 2524 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:53:40.387570 1212 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35805
I20250901 01:53:40.391285 2563 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" instance_seqno: 1756691596380664) as {username='slave'} at 127.0.104.2:53553; Asking this server to re-register.
I20250901 01:53:40.392437 1212 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:40.395013 2563 ts_manager.cc:194] Registered new tserver with Master: e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:40.607656 1069 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35805
I20250901 01:53:40.613255 2562 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" instance_seqno: 1756691595802033) as {username='slave'} at 127.0.104.1:36595; Asking this server to re-register.
I20250901 01:53:40.614418 1069 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:40.617250 2563 ts_manager.cc:194] Registered new tserver with Master: b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:40.618206 1352 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35805
I20250901 01:53:40.620249 2563 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ad2ba2450480454388723cd729cc438e" instance_seqno: 1756691596941203) as {username='slave'} at 127.0.104.3:33309; Asking this server to re-register.
I20250901 01:53:40.621140 1352 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:40.623018 2563 ts_manager.cc:194] Registered new tserver with Master: ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:40.750559 2602 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:40.750824 2602 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:40.752771 2602 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:40.753405 2240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 8 candidate_status { last_received { term: 7 index: 100 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f" is_pre_election: true
W20250901 01:53:40.753551 2550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:40.753686 2240 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c35f5a36eeb4a93aecdd71e2d1c3a21 in term 7.
I20250901 01:53:40.754398 2549 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, b557230fc3b142c4bf5f6f58d241905f; no voters:
I20250901 01:53:40.754882 2602 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250901 01:53:40.755055 2602 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:40.755203 2602 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 7 FOLLOWER]: Advancing to term 8
W20250901 01:53:40.755417 2550 leader_election.cc:336] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:40.758949 2602 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:40.759616 2602 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 8 election: Requested vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:40.760234 2240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 8 candidate_status { last_received { term: 7 index: 100 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f"
I20250901 01:53:40.760468 2240 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 7 FOLLOWER]: Advancing to term 8
W20250901 01:53:40.761265 2550 leader_election.cc:336] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:40.762867 2240 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c35f5a36eeb4a93aecdd71e2d1c3a21 in term 8.
I20250901 01:53:40.763423 2549 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, b557230fc3b142c4bf5f6f58d241905f; no voters: 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:40.763813 2602 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 8 FOLLOWER]: Leader election won for term 8
I20250901 01:53:40.764370 2602 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 8 LEADER]: Becoming Leader. State: Replica: 2c35f5a36eeb4a93aecdd71e2d1c3a21, State: Running, Role: LEADER
I20250901 01:53:40.764930 2602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 7.100, Last appended by leader: 100, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:40.767553 2629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c35f5a36eeb4a93aecdd71e2d1c3a21. Latest consensus state: current_term: 8 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:40.767802 2629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: LEADER
I20250901 01:53:40.768425 2631 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 01:53:40.771512 2631 catalog_manager.cc:671] Loaded metadata for table default.table_bb66c7b604104254a48d435595b83fc4 [id=267f90dc52214e7986546d0e99a7f23a]
I20250901 01:53:40.771992 2631 catalog_manager.cc:671] Loaded metadata for table default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19]
I20250901 01:53:40.772390 2631 catalog_manager.cc:671] Loaded metadata for table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]
I20250901 01:53:40.772782 2631 catalog_manager.cc:671] Loaded metadata for table default.table_8082166db25745e78737b640063b3219 [id=37fc4ed840564a8780eeb9847b373712]
I20250901 01:53:40.773110 2631 catalog_manager.cc:671] Loaded metadata for table default.table_8f21a26e011c4ed1b1ae7cb364e09195 [id=39123cfe4f62420681f8d679d2d376b8]
I20250901 01:53:40.773448 2631 catalog_manager.cc:671] Loaded metadata for table default.table_9f3451ed25714464ad16b3e94214d0e3 [id=5575a859d470414e9a1bf99d5738fc3a]
I20250901 01:53:40.773890 2631 catalog_manager.cc:671] Loaded metadata for table default.table_da487bf0d57f439bb9da2ec57880a73b [id=7185f28e50df4fb9a4bf41cc5b73169d]
I20250901 01:53:40.774295 2631 catalog_manager.cc:671] Loaded metadata for table default.table_167322850c734684b896b875526a5c6e [id=7a69995eebe446629422a7fe0924e58a]
I20250901 01:53:40.774618 2631 catalog_manager.cc:671] Loaded metadata for table default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d]
I20250901 01:53:40.774972 2631 catalog_manager.cc:671] Loaded metadata for table default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98]
I20250901 01:53:40.775283 2631 catalog_manager.cc:671] Loaded metadata for table default.table_1ff7214e853748e58232b9bea0cb6912 [id=af5c975082de48cfa7295b2d8bd16635]
I20250901 01:53:40.775629 2631 catalog_manager.cc:671] Loaded metadata for table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647]
I20250901 01:53:40.775956 2631 catalog_manager.cc:671] Loaded metadata for table default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9]
I20250901 01:53:40.776283 2631 catalog_manager.cc:671] Loaded metadata for table default.table_facefcd0049d44a2b465d8315527aa08 [id=d0ce834bce8e4a3fa795fb9e3b6b06a7]
I20250901 01:53:40.776633 2631 catalog_manager.cc:671] Loaded metadata for table default.table_c95fbf8fa7a64aa18002f1e268cdea68 [id=d830537abb3c488ca7b925a8a491432a]
I20250901 01:53:40.777149 2631 catalog_manager.cc:671] Loaded metadata for table default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5]
I20250901 01:53:40.777487 2631 catalog_manager.cc:671] Loaded metadata for table default.table_99b272393a1b40b4ae2d3bbaedc9ea60 [id=f868207bffb24b6795d0f636974d231d]
I20250901 01:53:40.779726 2631 tablet_loader.cc:96] loaded metadata for tablet 089fe953bba0403db288afefaf750c10 (table default.table_99b272393a1b40b4ae2d3bbaedc9ea60 [id=f868207bffb24b6795d0f636974d231d])
I20250901 01:53:40.780079 2631 tablet_loader.cc:96] loaded metadata for tablet 0e88f2701be74589b1337bd6f4ef87db (table default.table_da487bf0d57f439bb9da2ec57880a73b [id=7185f28e50df4fb9a4bf41cc5b73169d])
I20250901 01:53:40.780649 2631 tablet_loader.cc:96] loaded metadata for tablet 3733efc4223d49a28f69cb598bdf7c94 (table default.table_facefcd0049d44a2b465d8315527aa08 [id=d0ce834bce8e4a3fa795fb9e3b6b06a7])
I20250901 01:53:40.781288 2631 tablet_loader.cc:96] loaded metadata for tablet 5274573577374b47b99fa310a5ca83bb (table default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19])
I20250901 01:53:40.781581 2631 tablet_loader.cc:96] loaded metadata for tablet 586fbc1ef054435aaaea9ae961f15115 (table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647])
I20250901 01:53:40.782119 2631 tablet_loader.cc:96] loaded metadata for tablet 80d0e50666a34df39a4cc09977effe5c (table default.table_167322850c734684b896b875526a5c6e [id=7a69995eebe446629422a7fe0924e58a])
I20250901 01:53:40.782476 2631 tablet_loader.cc:96] loaded metadata for tablet 83b971ea891840229df1f7ed3e0d1d92 (table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd])
I20250901 01:53:40.782893 2631 tablet_loader.cc:96] loaded metadata for tablet 8dd9f23bffa543a490c018108ad41051 (table default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98])
I20250901 01:53:40.783174 2631 tablet_loader.cc:96] loaded metadata for tablet 8f913036f34345fb8b20b3c9f431979e (table default.table_8f21a26e011c4ed1b1ae7cb364e09195 [id=39123cfe4f62420681f8d679d2d376b8])
I20250901 01:53:40.783891 2631 tablet_loader.cc:96] loaded metadata for tablet a6af5a07f3ca4fe4a8808b913551c091 (table default.table_1ff7214e853748e58232b9bea0cb6912 [id=af5c975082de48cfa7295b2d8bd16635])
I20250901 01:53:40.784328 2631 tablet_loader.cc:96] loaded metadata for tablet ad4ea5cf599f4719960fdf0b2e6fc054 (table default.table_9f3451ed25714464ad16b3e94214d0e3 [id=5575a859d470414e9a1bf99d5738fc3a])
I20250901 01:53:40.784683 2631 tablet_loader.cc:96] loaded metadata for tablet b0064e7090a54fb18ee9a761ddec2c83 (table default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9])
I20250901 01:53:40.784986 2631 tablet_loader.cc:96] loaded metadata for tablet b302bfaed5104909a1d18f7529398348 (table default.table_8082166db25745e78737b640063b3219 [id=37fc4ed840564a8780eeb9847b373712])
I20250901 01:53:40.786362 2631 tablet_loader.cc:96] loaded metadata for tablet d8709bb983864d9f9e65eaf8f21a13f8 (table default.table_c95fbf8fa7a64aa18002f1e268cdea68 [id=d830537abb3c488ca7b925a8a491432a])
I20250901 01:53:40.786736 2631 tablet_loader.cc:96] loaded metadata for tablet de13eb65b0df4b8ebc049ff753123d53 (table default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5])
I20250901 01:53:40.787071 2631 tablet_loader.cc:96] loaded metadata for tablet debd885f996642faa60cfa5af0283b5f (table default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d])
I20250901 01:53:40.787400 2631 tablet_loader.cc:96] loaded metadata for tablet e4df70f8191e4115991d42178cb67bd7 (table default.table_bb66c7b604104254a48d435595b83fc4 [id=267f90dc52214e7986546d0e99a7f23a])
I20250901 01:53:40.787875 2631 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 01:53:40.789297 2631 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:40.789423 2631 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 01:53:40.791225 2631 catalog_manager.cc:1506] Loading token signing keys...
I20250901 01:53:40.792523 2631 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Loaded TSK: 0
I20250901 01:53:40.793579 2631 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 01:53:40.794914 2631 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 01:53:40.796197 2631 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 22
I20250901 01:53:40.842315 2563 master_service.cc:948] ReplaceTablet: received request to replace tablet ad4ea5cf599f4719960fdf0b2e6fc054 from {username='slave'} at 127.0.0.1:42216
I20250901 01:53:40.842324 2562 master_service.cc:948] ReplaceTablet: received request to replace tablet 5274573577374b47b99fa310a5ca83bb from {username='slave'} at 127.0.0.1:42216
I20250901 01:53:40.843026 2560 master_service.cc:948] ReplaceTablet: received request to replace tablet 5274573577374b47b99fa310a5ca83bb from {username='slave'} at 127.0.0.1:42216
I20250901 01:53:40.843427 2561 master_service.cc:948] ReplaceTablet: received request to replace tablet 83b971ea891840229df1f7ed3e0d1d92 from {username='slave'} at 127.0.0.1:42216
I20250901 01:53:40.843509 2558 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42216:
table { table_name: "default.Table_5d5e7ea4ac7d4e5f9de3237058d11e80" } modify_external_catalogs: true
I20250901 01:53:40.843813 2558 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42216:
table { table_name: "default.Table_5d5e7ea4ac7d4e5f9de3237058d11e80" } modify_external_catalogs: true
I20250901 01:53:40.844568 2559 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42216:
table { table_name: "default.Table_c95fbf8fa7a64aa18002f1e268cdea68" } new_table_name: "default.Table_9afa854fa1c148929b0ede8977435270" modify_external_catalogs: true
I20250901 01:53:40.845547 2554 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42216:
table { table_name: "default.Table_1ff7214e853748e58232b9bea0cb6912" } new_table_name: "default.Table_1b2b89cc1bc2459b80fd9de967ebd9a1" modify_external_catalogs: true
I20250901 01:53:40.845881 2557 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42216:
table { table_name: "default.Table_61dfd07a8f784854b5d9e104e010daaa" } new_table_name: "default.Table_757f6c2777014fe1b0686fc1405b4b5f" modify_external_catalogs: true
I20250901 01:53:40.846710 2556 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42216:
table { table_name: "default.Table_d1c05496169b472fa9b021013c4afd01" } modify_external_catalogs: true
I20250901 01:53:40.847405 2556 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42216:
table { table_name: "default.Table_d1c05496169b472fa9b021013c4afd01" } modify_external_catalogs: true
I20250901 01:53:40.847414 2555 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42216:
table { table_name: "default.Table_7e006da3e6d24d309af545afeac11ca4" } new_table_name: "default.Table_746437fc94f24e50977140cddc823eaf" modify_external_catalogs: true
W20250901 01:53:40.852665 2550 consensus_peers.cc:489] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 -> Peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Couldn't send request to peer 835a442fbc4c48eb9148ca677e410a59. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20250901 01:53:40.853062 2615 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 5 out of 13 received events: NotificationEvent(eventId=23, eventTime=1756691601, eventType=CREATE_TABLE, dbName=default, tableName=table_98a9d1400149426bab73e31aea263f5b, message={"server":"","servicePrincipal":"","db":"default","table":"table_98a9d1400149426bab73e31aea263f5b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_98a9d1400149426bab73e31aea263f5b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691601},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_98a9d1400149426bab73e31aea263f5b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_98a9d1400149426bab73e31aea263f5b\",\"kudu.table_id\":\"49527ae170f04658b8b1eb01b07ffd04\",\"transient_lastDdlTime\":\"1756691601\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1756691601, eventType=ALTER_TABLE, dbName=default, tableName=table_757f6c2777014fe1b0686fc1405b4b5f, message={"server":"","servicePrincipal":"","db":"default","table":"table_61dfd07a8f784854b5d9e104e010daaa","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_61dfd07a8f784854b5d9e104e010daaa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691599},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_61dfd07a8f784854b5d9e104e010daaa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_61dfd07a8f784854b5d9e104e010daaa\",\"kudu.table_id\":\"ae0da2fb2d1542308ccb9a95247c1d98\",\"transient_lastDdlTime\":\"1756691599\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_757f6c2777014fe1b0686fc1405b4b5f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691599},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_757f6c2777014fe1b0686fc1405b4b5f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_757f6c2777014fe1b0686fc1405b4b5f\",\"kudu.table_id\":\"ae0da2fb2d1542308ccb9a95247c1d98\",\"transient_lastDdlTime\":\"1756691599\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_c2864548fe1a4719b73cf55d0933e538, message={"server":"","servicePrincipal":"","db":"default","table":"table_c2864548fe1a4719b73cf55d0933e538","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c2864548fe1a4719b73cf55d0933e538\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691598},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_c2864548fe1a4719b73cf55d0933e538\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c2864548fe1a4719b73cf55d0933e538\",\"kudu.table_id\":\"31d721fb8b204568a0190dda6def01cd\",\"transient_lastDdlTime\":\"1756691598\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_56fde480a00a4e62b84eedba22b0bade, message={"server":"","servicePrincipal":"","db":"default","table":"table_56fde480a00a4e62b84eedba22b0bade","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_56fde480a00a4e62b84eedba22b0bade\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691601},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_56fde480a00a4e62b84eedba22b0bade\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_56fde480a00a4e62b84eedba22b0bade\",\"kudu.table_id\":\"181eeac3e2394bcbaf3071e3fad223d8\",\"transient_lastDdlTime\":\"1756691601\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_7722e49197be4ca9b9c7e67052f07c4b, message={"server":"","servicePrincipal":"","db":"default","table":"table_7722e49197be4ca9b9c7e67052f07c4b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_7722e49197be4ca9b9c7e67052f07c4b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691598},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_7722e49197be4ca9b9c7e67052f07c4b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7722e49197be4ca9b9c7e67052f07c4b\",\"kudu.table_id\":\"2e950743029d48a49f05207cd5faee19\",\"transient_lastDdlTime\":\"1756691598\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f7c73209dcd google::LogMessage::Fail() at ??:0
@ 0x7f7c7320db93 google::LogMessage::SendToLog() at ??:0
I20250901 01:53:40.859211 2240 raft_consensus.cc:1273] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Refusing update from remote peer 2c35f5a36eeb4a93aecdd71e2d1c3a21: Log matching property violated. Preceding OpId in replica: term: 7 index: 100. Preceding OpId from leader: term: 8 index: 103. (index mismatch)
@ 0x7f7c732097cc google::LogMessage::Flush() at ??:0
I20250901 01:53:40.860221 2602 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
@ 0x7f7c73209a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f7c8963e794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f7c8963d041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f7c89644cbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f7c89644b39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250901 01:53:40.889603 2635 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c35f5a36eeb4a93aecdd71e2d1c3a21. Latest consensus state: current_term: 8 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:40.889952 2635 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:40.892498 2632 mvcc.cc:204] Tried to move back new op lower bound from 7195408878995861504 to 7195408878657191936. Current Snapshot: MvccSnapshot[applied={T|T < 7195408878995861504}]
I20250901 01:53:40.895851 2562 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Sending DeleteTablet for 3 replicas of tablet 5274573577374b47b99fa310a5ca83bb
@ 0x7f7c75f66cc9 kudu::Thread::SuperviseThread() at ??:0
I20250901 01:53:40.897956 2562 catalog_manager.cc:6652] ReplaceTablet: tablet 5274573577374b47b99fa310a5ca83bb deleted and replaced by tablet 45fc5cc524a0438394ed533406785f45
@ 0x7f7c793886db start_thread at ??:0
@ 0x7f7c720ba71f clone at ??:0
I20250901 01:53:40.917399 2635 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:40.917639 2635 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
W20250901 01:53:41.096256 2647 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 01:53:40.900573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.0.104.3:42927)
0901 01:53:40.913674 (+ 13101us) server_negotiation.cc:206] Beginning negotiation
0901 01:53:40.913681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 01:53:41.095950 (+182269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37762: BlockingRecv error: recv got EOF from 127.0.0.1:37762 (error 108)
Metrics: {"server-negotiator.queue_time_us":12956,"thread_start_us":14221,"threads_started":1}
W20250901 01:53:41.096273 2646 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 01:53:40.900288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.0.104.2:35245)
0901 01:53:40.913279 (+ 12991us) server_negotiation.cc:206] Beginning negotiation
0901 01:53:40.913290 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0901 01:53:41.095910 (+182620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57282: BlockingRecv error: recv got EOF from 127.0.0.1:57282 (error 108)
Metrics: {"server-negotiator.queue_time_us":12800,"thread_start_us":14202,"threads_started":1}
W20250901 01:53:41.096666 2215 connection.cc:537] server connection from 127.0.0.1:44074 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:41.096982 1100 connection.cc:537] client connection to 127.0.104.62:35805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:41.097179 2643 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 01:53:40.899237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.0.104.1:45315)
0901 01:53:40.914117 (+ 14880us) server_negotiation.cc:206] Beginning negotiation
0901 01:53:40.914124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 01:53:40.914171 (+ 47us) server_negotiation.cc:407] Connection header received
0901 01:53:40.914273 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 01:53:40.914280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 01:53:40.914385 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 01:53:40.914524 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 01:53:41.095968 (+181444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38700: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":14744,"thread_start_us":15099,"threads_started":1}
W20250901 01:53:41.098277 1362 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.62:35805)
W20250901 01:53:41.098542 1362 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.62:35805)
W20250901 01:53:41.098781 1362 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.62:35805)
W20250901 01:53:41.099658 1362 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.62:35805)
I20250901 01:53:41.395366 2635 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:41.395557 2635 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:41.396302 2635 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:41.399670 2213 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:41.399952 2216 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:41.400118 2216 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:41.400434 2635 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250901 01:53:42.007993 2671 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:42.008198 2671 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:42.008891 2671 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:42.010919 2216 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:53:42.011165 2213 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:53:42.011276 2213 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:42.011535 2671 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250901 01:53:42.657253 2192 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:53:42.657467 2192 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:53:42.839761 2684 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:42.840018 2684 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:42.840850 2684 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:42.842941 2216 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:53:42.843144 2213 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:53:42.843262 2213 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:42.843569 2684 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250901 01:53:43.387516 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1fHXBS/build/asan/bin/kudu
/tmp/dist-test-task1fHXBS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35805
--webserver_interface=127.0.104.62
--webserver_port=32963
--builtin_ntp_servers=127.0.104.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250901 01:53:43.393483 2690 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:43.393684 2690 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:43.400583 2690 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:43.400947 2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20250901 01:53:43.407090 2213 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:43.407341 2216 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:43.407491 2216 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:43.407770 2690 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250901 01:53:43.759358 2689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:53:43.759737 2689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:53:43.759871 2689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 01:53:43.771100 2689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 01:53:43.771221 2689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:53:43.771323 2689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 01:53:43.771392 2689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 01:53:43.782071 2689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=32963
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
I20250901 01:53:43.783843 2689 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:53:43.785676 2689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 01:53:43.794587 2707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:43.795064 2709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:43.794626 2706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:43.796216 2689 server_base.cc:1047] running on GCE node
I20250901 01:53:43.797103 2689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:53:43.798792 2689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:53:43.800060 2689 hybrid_clock.cc:648] HybridClock initialized: now 1756691623800002 us; error 50 us; skew 500 ppm
I20250901 01:53:43.800480 2689 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:53:43.803700 2689 webserver.cc:480] Webserver started at http://127.0.104.62:32963/ using document root <none> and password file <none>
I20250901 01:53:43.804553 2689 fs_manager.cc:362] Metadata directory not provided
I20250901 01:53:43.804697 2689 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 01:53:43.810734 2689 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20250901 01:53:43.813750 2715 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:43.815338 2689 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250901 01:53:43.815524 2689 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data,/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21"
format_stamp: "Formatted at 2025-09-01 01:53:13 on dist-test-slave-r4vn"
I20250901 01:53:43.816264 2689 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:53:43.845250 2689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 01:53:43.846259 2689 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:53:43.846576 2689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:53:43.867463 2689 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35805
I20250901 01:53:43.867492 2766 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35805 every 8 connection(s)
I20250901 01:53:43.869293 2689 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-0/data/info.pb
I20250901 01:53:43.871779 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1fHXBS/build/asan/bin/kudu as pid 2689
I20250901 01:53:43.878863 2767 sys_catalog.cc:263] Verifying existing consensus state
I20250901 01:53:43.884275 2767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Bootstrap starting.
I20250901 01:53:43.905246 2767 log.cc:826] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Log is configured to *not* fsync() on all Append() calls
I20250901 01:53:43.972736 1212 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35805
I20250901 01:53:43.985822 2767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=104 overwritten=0 applied=102 ignored=0} inserts{seen=66 ignored=0} mutations{seen=79 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250901 01:53:43.986515 2767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Bootstrap complete.
I20250901 01:53:43.991899 2767 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 8 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:43.995011 2767 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c35f5a36eeb4a93aecdd71e2d1c3a21, State: Initialized, Role: FOLLOWER
I20250901 01:53:43.995784 2767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 102, Last appended: 8.104, Last appended by leader: 104, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:43.996565 2773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:43.996824 2773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:43.998008 2767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 01:53:44.019407 2787 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:44.019608 2787 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: loading cluster ID for follower catalog manager: success
I20250901 01:53:44.026240 2787 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: acquiring CA information for follower catalog manager: success
I20250901 01:53:44.030263 2787 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 01:53:44.034510 2689 master_runner.cc:186] Error getting master registration for 127.0.104.60:36985: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111),
I20250901 01:53:44.038235 2689 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:53:44.140065 1352 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35805
I20250901 01:53:44.142889 2731 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ad2ba2450480454388723cd729cc438e" instance_seqno: 1756691596941203) as {username='slave'} at 127.0.104.3:52323; Asking this server to re-register.
I20250901 01:53:44.143853 1352 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:44.146274 2731 ts_manager.cc:194] Registered new tserver with Master: ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:44.168820 1069 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35805
I20250901 01:53:44.170894 2731 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" instance_seqno: 1756691595802033) as {username='slave'} at 127.0.104.1:60663; Asking this server to re-register.
I20250901 01:53:44.171557 1069 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:44.173389 2731 ts_manager.cc:194] Registered new tserver with Master: b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:44.477476 1212 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35805
I20250901 01:53:44.479281 2731 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" instance_seqno: 1756691596380664) as {username='slave'} at 127.0.104.2:40767; Asking this server to re-register.
I20250901 01:53:44.479967 1212 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:44.481416 2731 ts_manager.cc:194] Registered new tserver with Master: e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:44.553576 2794 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:44.553982 2794 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:44.556576 2794 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:44.557108 2240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 9 candidate_status { last_received { term: 8 index: 104 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f" is_pre_election: true
I20250901 01:53:44.557361 2240 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c35f5a36eeb4a93aecdd71e2d1c3a21 in term 8.
W20250901 01:53:44.558212 2719 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:44.558275 2718 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, b557230fc3b142c4bf5f6f58d241905f; no voters:
I20250901 01:53:44.558735 2794 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250901 01:53:44.558925 2794 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:53:44.559052 2794 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 8 FOLLOWER]: Advancing to term 9
W20250901 01:53:44.562237 2719 leader_election.cc:336] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:44.564357 2794 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:44.565038 2794 leader_election.cc:290] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 9 election: Requested vote from peers b557230fc3b142c4bf5f6f58d241905f (127.0.104.61:33025), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
I20250901 01:53:44.565647 2240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" candidate_term: 9 candidate_status { last_received { term: 8 index: 104 } } ignore_live_leader: false dest_uuid: "b557230fc3b142c4bf5f6f58d241905f"
I20250901 01:53:44.565883 2240 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 8 FOLLOWER]: Advancing to term 9
W20250901 01:53:44.566661 2719 leader_election.cc:336] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:44.568064 2240 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c35f5a36eeb4a93aecdd71e2d1c3a21 in term 9.
I20250901 01:53:44.568502 2718 leader_election.cc:304] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, b557230fc3b142c4bf5f6f58d241905f; no voters: 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:44.568826 2794 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 9 FOLLOWER]: Leader election won for term 9
I20250901 01:53:44.569208 2794 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [term 9 LEADER]: Becoming Leader. State: Replica: 2c35f5a36eeb4a93aecdd71e2d1c3a21, State: Running, Role: LEADER
I20250901 01:53:44.569662 2794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 8.104, Last appended by leader: 104, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:44.572434 2797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c35f5a36eeb4a93aecdd71e2d1c3a21. Latest consensus state: current_term: 9 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:44.572695 2797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: LEADER
W20250901 01:53:44.954653 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 11: this message will repeat every 5th retry.
W20250901 01:53:44.971827 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 11: this message will repeat every 5th retry.
I20250901 01:53:45.014189 2240 raft_consensus.cc:1273] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Refusing update from remote peer 2c35f5a36eeb4a93aecdd71e2d1c3a21: Log matching property violated. Preceding OpId in replica: term: 8 index: 104. Preceding OpId from leader: term: 9 index: 105. (index mismatch)
I20250901 01:53:45.015059 2797 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 103, Time since last communication: 0.000s
I20250901 01:53:45.017477 2803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c35f5a36eeb4a93aecdd71e2d1c3a21. Latest consensus state: current_term: 9 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:45.017767 2803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:45.024032 2803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:45.024272 2803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:45.025360 2797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:45.025604 2797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 [sys.catalog]: This master's current role is: LEADER
W20250901 01:53:45.026254 2719 consensus_peers.cc:489] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21 -> Peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Couldn't send request to peer 835a442fbc4c48eb9148ca677e410a59. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:53:45.089978 2799 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 01:53:45.092974 2799 catalog_manager.cc:671] Loaded metadata for table default.table_bb66c7b604104254a48d435595b83fc4 [id=267f90dc52214e7986546d0e99a7f23a]
I20250901 01:53:45.093478 2799 catalog_manager.cc:671] Loaded metadata for table default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19]
I20250901 01:53:45.093854 2799 catalog_manager.cc:671] Loaded metadata for table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd]
I20250901 01:53:45.094233 2799 catalog_manager.cc:671] Loaded metadata for table default.table_8082166db25745e78737b640063b3219 [id=37fc4ed840564a8780eeb9847b373712]
I20250901 01:53:45.094570 2799 catalog_manager.cc:671] Loaded metadata for table default.table_8f21a26e011c4ed1b1ae7cb364e09195 [id=39123cfe4f62420681f8d679d2d376b8]
I20250901 01:53:45.094913 2799 catalog_manager.cc:671] Loaded metadata for table default.table_9f3451ed25714464ad16b3e94214d0e3 [id=5575a859d470414e9a1bf99d5738fc3a]
I20250901 01:53:45.095376 2799 catalog_manager.cc:671] Loaded metadata for table default.table_da487bf0d57f439bb9da2ec57880a73b [id=7185f28e50df4fb9a4bf41cc5b73169d]
I20250901 01:53:45.095708 2799 catalog_manager.cc:671] Loaded metadata for table default.table_167322850c734684b896b875526a5c6e [id=7a69995eebe446629422a7fe0924e58a]
I20250901 01:53:45.096029 2799 catalog_manager.cc:671] Loaded metadata for table default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d]
I20250901 01:53:45.096347 2799 catalog_manager.cc:671] Loaded metadata for table default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98]
I20250901 01:53:45.096652 2799 catalog_manager.cc:671] Loaded metadata for table default.table_1ff7214e853748e58232b9bea0cb6912 [id=af5c975082de48cfa7295b2d8bd16635]
I20250901 01:53:45.096982 2799 catalog_manager.cc:671] Loaded metadata for table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647]
I20250901 01:53:45.097337 2799 catalog_manager.cc:671] Loaded metadata for table default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9]
I20250901 01:53:45.097697 2799 catalog_manager.cc:671] Loaded metadata for table default.table_facefcd0049d44a2b465d8315527aa08 [id=d0ce834bce8e4a3fa795fb9e3b6b06a7]
I20250901 01:53:45.098019 2799 catalog_manager.cc:671] Loaded metadata for table default.table_c95fbf8fa7a64aa18002f1e268cdea68 [id=d830537abb3c488ca7b925a8a491432a]
I20250901 01:53:45.098511 2799 catalog_manager.cc:671] Loaded metadata for table default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5]
I20250901 01:53:45.098839 2799 catalog_manager.cc:671] Loaded metadata for table default.table_99b272393a1b40b4ae2d3bbaedc9ea60 [id=f868207bffb24b6795d0f636974d231d]
I20250901 01:53:45.101177 2799 tablet_loader.cc:96] loaded metadata for tablet 089fe953bba0403db288afefaf750c10 (table default.table_99b272393a1b40b4ae2d3bbaedc9ea60 [id=f868207bffb24b6795d0f636974d231d])
I20250901 01:53:45.101631 2799 tablet_loader.cc:96] loaded metadata for tablet 0e88f2701be74589b1337bd6f4ef87db (table default.table_da487bf0d57f439bb9da2ec57880a73b [id=7185f28e50df4fb9a4bf41cc5b73169d])
I20250901 01:53:45.102291 2799 tablet_loader.cc:96] loaded metadata for tablet 3733efc4223d49a28f69cb598bdf7c94 (table default.table_facefcd0049d44a2b465d8315527aa08 [id=d0ce834bce8e4a3fa795fb9e3b6b06a7])
I20250901 01:53:45.102794 2799 tablet_loader.cc:96] loaded metadata for tablet 45fc5cc524a0438394ed533406785f45 (table default.table_7722e49197be4ca9b9c7e67052f07c4b [id=2e950743029d48a49f05207cd5faee19])
I20250901 01:53:45.103497 2799 tablet_loader.cc:96] loaded metadata for tablet 586fbc1ef054435aaaea9ae961f15115 (table default.table_d1c05496169b472fa9b021013c4afd01 [id=b449add8498f4d1fbb05b28cc933c647])
I20250901 01:53:45.104064 2799 tablet_loader.cc:96] loaded metadata for tablet 6b68b9d93a0345338bd6505f2dd39887 (table default.table_c2864548fe1a4719b73cf55d0933e538 [id=31d721fb8b204568a0190dda6def01cd])
I20250901 01:53:45.104382 2799 tablet_loader.cc:96] loaded metadata for tablet 80d0e50666a34df39a4cc09977effe5c (table default.table_167322850c734684b896b875526a5c6e [id=7a69995eebe446629422a7fe0924e58a])
I20250901 01:53:45.104997 2799 tablet_loader.cc:96] loaded metadata for tablet 8dd9f23bffa543a490c018108ad41051 (table default.table_61dfd07a8f784854b5d9e104e010daaa [id=ae0da2fb2d1542308ccb9a95247c1d98])
I20250901 01:53:45.105301 2799 tablet_loader.cc:96] loaded metadata for tablet 8f913036f34345fb8b20b3c9f431979e (table default.table_8f21a26e011c4ed1b1ae7cb364e09195 [id=39123cfe4f62420681f8d679d2d376b8])
I20250901 01:53:45.105579 2799 tablet_loader.cc:96] loaded metadata for tablet 9c12fd4d88b4476494a8fd27331fbc4c (table default.table_9f3451ed25714464ad16b3e94214d0e3 [id=5575a859d470414e9a1bf99d5738fc3a])
I20250901 01:53:45.106484 2799 tablet_loader.cc:96] loaded metadata for tablet a6af5a07f3ca4fe4a8808b913551c091 (table default.table_1ff7214e853748e58232b9bea0cb6912 [id=af5c975082de48cfa7295b2d8bd16635])
I20250901 01:53:45.107182 2799 tablet_loader.cc:96] loaded metadata for tablet b0064e7090a54fb18ee9a761ddec2c83 (table default.table_5178a246373c4a26a1a2e11a77c7e56c [id=bffe8fa63e6045b98694851a029bc0d9])
I20250901 01:53:45.107497 2799 tablet_loader.cc:96] loaded metadata for tablet b302bfaed5104909a1d18f7529398348 (table default.table_8082166db25745e78737b640063b3219 [id=37fc4ed840564a8780eeb9847b373712])
I20250901 01:53:45.108453 2799 tablet_loader.cc:96] loaded metadata for tablet d8709bb983864d9f9e65eaf8f21a13f8 (table default.table_c95fbf8fa7a64aa18002f1e268cdea68 [id=d830537abb3c488ca7b925a8a491432a])
I20250901 01:53:45.108752 2799 tablet_loader.cc:96] loaded metadata for tablet de13eb65b0df4b8ebc049ff753123d53 (table default.table_7e006da3e6d24d309af545afeac11ca4 [id=f1db819ca90642eaac30e77fd5f5c8d5])
I20250901 01:53:45.109036 2799 tablet_loader.cc:96] loaded metadata for tablet debd885f996642faa60cfa5af0283b5f (table default.table_5d5e7ea4ac7d4e5f9de3237058d11e80 [id=8ee4e0d7bca3410b9967df0dacf8614d])
I20250901 01:53:45.109329 2799 tablet_loader.cc:96] loaded metadata for tablet e4df70f8191e4115991d42178cb67bd7 (table default.table_bb66c7b604104254a48d435595b83fc4 [id=267f90dc52214e7986546d0e99a7f23a])
I20250901 01:53:45.109792 2799 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 01:53:45.111164 2799 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:45.111250 2799 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 01:53:45.112864 2799 catalog_manager.cc:1506] Loading token signing keys...
I20250901 01:53:45.114064 2799 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2c35f5a36eeb4a93aecdd71e2d1c3a21: Loaded TSK: 0
I20250901 01:53:45.114993 2799 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 01:53:45.116185 2799 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 01:53:45.117429 2799 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 22
I20250901 01:53:45.132939 2731 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54870:
table { table_name: "default.Table_8f21a26e011c4ed1b1ae7cb364e09195" } modify_external_catalogs: true
I20250901 01:53:45.133255 2732 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54870:
name: "default.Table_0084f9e4b939410da1d13247891e6ac4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:45.133436 2728 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54870:
name: "default.Table_56fde480a00a4e62b84eedba22b0bade"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:45.133601 2730 master_service.cc:948] ReplaceTablet: received request to replace tablet 3f1db382c27e455cae9ddf28a79ada2b from {username='slave'} at 127.0.0.1:54870
I20250901 01:53:45.134342 2729 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54870:
name: "default.Table_98a9d1400149426bab73e31aea263f5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:45.134790 2724 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54870:
name: "default.Table_c1e876d5919f4c1f98a0ef9d1fcb9edc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:45.135154 2730 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54870:
table { table_name: "default.Table_c95fbf8fa7a64aa18002f1e268cdea68" } new_table_name: "default.Table_9afa854fa1c148929b0ede8977435270" modify_external_catalogs: true
I20250901 01:53:45.135056 2727 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54870:
name: "default.Table_ed3c253535084a3399b30083bb388f72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:45.135641 2723 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54870:
table { table_name: "default.Table_5d5e7ea4ac7d4e5f9de3237058d11e80" } modify_external_catalogs: true
I20250901 01:53:45.135913 2723 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54870:
table { table_name: "default.Table_5d5e7ea4ac7d4e5f9de3237058d11e80" } modify_external_catalogs: true
I20250901 01:53:45.134289 2726 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54870:
name: "default.Table_cbe4bb8385f54f7f9359f495d8c663a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:45.136034 2725 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54870:
name: "default.Table_f6c5cfd184ce474b883ce77bd225eb9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:53:45.133280 2731 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54870:
table { table_name: "default.Table_8f21a26e011c4ed1b1ae7cb364e09195" } modify_external_catalogs: true
F20250901 01:53:45.141058 2786 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 5 out of 13 received events: NotificationEvent(eventId=23, eventTime=1756691601, eventType=CREATE_TABLE, dbName=default, tableName=table_98a9d1400149426bab73e31aea263f5b, message={"server":"","servicePrincipal":"","db":"default","table":"table_98a9d1400149426bab73e31aea263f5b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_98a9d1400149426bab73e31aea263f5b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691601},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_98a9d1400149426bab73e31aea263f5b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_98a9d1400149426bab73e31aea263f5b\",\"kudu.table_id\":\"49527ae170f04658b8b1eb01b07ffd04\",\"transient_lastDdlTime\":\"1756691601\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=24, eventTime=1756691601, eventType=ALTER_TABLE, dbName=default, tableName=table_757f6c2777014fe1b0686fc1405b4b5f, message={"server":"","servicePrincipal":"","db":"default","table":"table_61dfd07a8f784854b5d9e104e010daaa","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_61dfd07a8f784854b5d9e104e010daaa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691599},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_61dfd07a8f784854b5d9e104e010daaa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_61dfd07a8f784854b5d9e104e010daaa\",\"kudu.table_id\":\"ae0da2fb2d1542308ccb9a95247c1d98\",\"transient_lastDdlTime\":\"1756691599\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_757f6c2777014fe1b0686fc1405b4b5f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691599},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_757f6c2777014fe1b0686fc1405b4b5f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_757f6c2777014fe1b0686fc1405b4b5f\",\"kudu.table_id\":\"ae0da2fb2d1542308ccb9a95247c1d98\",\"transient_lastDdlTime\":\"1756691599\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_c2864548fe1a4719b73cf55d0933e538, message={"server":"","servicePrincipal":"","db":"default","table":"table_c2864548fe1a4719b73cf55d0933e538","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c2864548fe1a4719b73cf55d0933e538\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691598},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_c2864548fe1a4719b73cf55d0933e538\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c2864548fe1a4719b73cf55d0933e538\",\"kudu.table_id\":\"31d721fb8b204568a0190dda6def01cd\",\"transient_lastDdlTime\":\"1756691598\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_56fde480a00a4e62b84eedba22b0bade, message={"server":"","servicePrincipal":"","db":"default","table":"table_56fde480a00a4e62b84eedba22b0bade","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_56fde480a00a4e62b84eedba22b0bade\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691601},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_56fde480a00a4e62b84eedba22b0bade\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_56fde480a00a4e62b84eedba22b0bade\",\"kudu.table_id\":\"181eeac3e2394bcbaf3071e3fad223d8\",\"transient_lastDdlTime\":\"1756691601\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1756691601, eventType=DROP_TABLE, dbName=default, tableName=table_7722e49197be4ca9b9c7e67052f07c4b, message={"server":"","servicePrincipal":"","db":"default","table":"table_7722e49197be4ca9b9c7e67052f07c4b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_7722e49197be4ca9b9c7e67052f07c4b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756691598},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/warehouse/table_7722e49197be4ca9b9c7e67052f07c4b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7722e49197be4ca9b9c7e67052f07c4b\",\"kudu.table_id\":\"2e950743029d48a49f05207cd5faee19\",\"transient_lastDdlTime\":\"1756691598\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"054914dc0c3a43f3bb1d17a9b313e30a\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756691601}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f5ffc321dcd google::LogMessage::Fail() at ??:0
@ 0x7f5ffc325b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f5ffc3217cc google::LogMessage::Flush() at ??:0
@ 0x7f5ffc321a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f6012756794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f6012755041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f601275ccbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f601275cb39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f5fff07ecc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f60024a06db start_thread at ??:0
@ 0x7f5ffb1d271f clone at ??:0
W20250901 01:53:45.356244 1362 connection.cc:537] client connection to 127.0.104.62:35805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:53:45.356850 1362 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.62:35805)
W20250901 01:53:45.357102 1362 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.62:35805)
W20250901 01:53:45.357493 1362 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.62:35805)
W20250901 01:53:45.357746 1362 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.62:35805)
I20250901 01:53:45.520906 2803 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:45.521092 2803 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:45.521757 2803 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:45.524150 2213 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:45.524307 2216 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:45.524453 2216 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:45.524725 2803 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:53:46.103595 2837 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2c35f5a36eeb4a93aecdd71e2d1c3a21)
I20250901 01:53:46.103787 2837 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:46.104416 2837 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:46.107077 2216 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:53:46.107246 2213 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:53:46.107358 2213 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:46.107586 2837 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:53:46.467060 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task1fHXBS/build/asan/bin/kudu pid 2192
I20250901 01:53:46.473837 2260 generic_service.cc:196] Checking for leaks (request via RPC)
I20250901 01:53:46.594219 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task1fHXBS/build/asan/bin/kudu with pid 2192
I20250901 01:53:47.314244 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-task1fHXBS/build/asan/bin/kudu
/tmp/dist-test-task1fHXBS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:33025
--webserver_interface=127.0.104.61
--webserver_port=34857
--builtin_ntp_servers=127.0.104.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250901 01:53:47.670506 2853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:53:47.670866 2853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:53:47.670956 2853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 01:53:47.680645 2853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 01:53:47.680757 2853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:53:47.680815 2853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 01:53:47.680887 2853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 01:53:47.691411 2853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:33327
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=34857
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
I20250901 01:53:47.693100 2853 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:53:47.695008 2853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 01:53:47.704826 2863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:47.704887 2860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:53:47.704909 2861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:53:47.705307 2853 server_base.cc:1047] running on GCE node
I20250901 01:53:47.706555 2853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:53:47.708160 2853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:53:47.709393 2853 hybrid_clock.cc:648] HybridClock initialized: now 1756691627709328 us; error 54 us; skew 500 ppm
I20250901 01:53:47.709792 2853 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:53:47.712257 2853 webserver.cc:480] Webserver started at http://127.0.104.61:34857/ using document root <none> and password file <none>
I20250901 01:53:47.713198 2853 fs_manager.cc:362] Metadata directory not provided
I20250901 01:53:47.713326 2853 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 01:53:47.719322 2853 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20250901 01:53:47.722720 2869 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:53:47.724447 2853 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250901 01:53:47.724656 2853 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data,/tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
uuid: "b557230fc3b142c4bf5f6f58d241905f"
format_stamp: "Formatted at 2025-09-01 01:53:13 on dist-test-slave-r4vn"
I20250901 01:53:47.725414 2853 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:53:47.749305 2853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 01:53:47.750314 2853 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:53:47.750614 2853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:53:47.772759 2853 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:33025
I20250901 01:53:47.772765 2920 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:33025 every 8 connection(s)
I20250901 01:53:47.774811 2853 server_base.cc:1179] Dumped server information to /tmp/dist-test-task1fHXBS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756691578137259-416-0/minicluster-data/master-1/data/info.pb
I20250901 01:53:47.782912 2921 sys_catalog.cc:263] Verifying existing consensus state
I20250901 01:53:47.784413 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-task1fHXBS/build/asan/bin/kudu as pid 2853
I20250901 01:53:47.787407 2921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Bootstrap starting.
I20250901 01:53:47.808248 2921 log.cc:826] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Log is configured to *not* fsync() on all Append() calls
I20250901 01:53:47.859313 1214 heartbeater.cc:344] Connected to a master server at 127.0.104.61:33025
I20250901 01:53:47.891325 2921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Bootstrap replayed 1/1 log segments. Stats: ops{read=105 overwritten=0 applied=105 ignored=0} inserts{seen=68 ignored=0} mutations{seen=81 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 01:53:47.891942 2921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: Bootstrap complete.
I20250901 01:53:47.897521 2921 raft_consensus.cc:357] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:47.898281 2921 raft_consensus.cc:738] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: b557230fc3b142c4bf5f6f58d241905f, State: Initialized, Role: FOLLOWER
I20250901 01:53:47.899010 2921 consensus_queue.cc:260] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 105, Last appended: 9.105, Last appended by leader: 105, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:47.899899 2927 sys_catalog.cc:455] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } } }
I20250901 01:53:47.900153 2927 sys_catalog.cc:458] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:53:47.901002 2921 sys_catalog.cc:564] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [sys.catalog]: configured and running, proceeding with master startup.
I20250901 01:53:47.927902 2941 catalog_manager.cc:1261] Loaded cluster ID: 054914dc0c3a43f3bb1d17a9b313e30a
I20250901 01:53:47.928004 2941 catalog_manager.cc:1554] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: loading cluster ID for follower catalog manager: success
I20250901 01:53:47.931948 2941 catalog_manager.cc:1576] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: acquiring CA information for follower catalog manager: success
I20250901 01:53:47.927381 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:53:47.933794 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:53:47.935323 2941 catalog_manager.cc:1604] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 01:53:47.994565 2886 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e3c774a4edf44d6098a2332e734967a5" instance_seqno: 1756691596380664) as {username='slave'} at 127.0.104.2:40175; Asking this server to re-register.
I20250901 01:53:47.995518 1214 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:47.998024 2886 ts_manager.cc:194] Registered new tserver with Master: e3c774a4edf44d6098a2332e734967a5 (127.0.104.2:35245)
I20250901 01:53:48.137894 1354 heartbeater.cc:344] Connected to a master server at 127.0.104.61:33025
I20250901 01:53:48.139928 2886 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ad2ba2450480454388723cd729cc438e" instance_seqno: 1756691596941203) as {username='slave'} at 127.0.104.3:45995; Asking this server to re-register.
I20250901 01:53:48.140702 1354 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:48.142968 2886 ts_manager.cc:194] Registered new tserver with Master: ad2ba2450480454388723cd729cc438e (127.0.104.3:42927)
I20250901 01:53:48.189005 1071 heartbeater.cc:344] Connected to a master server at 127.0.104.61:33025
I20250901 01:53:48.191097 2886 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b65b3154aefd49f7a37f70e9207e9853" instance_seqno: 1756691595802033) as {username='slave'} at 127.0.104.1:58045; Asking this server to re-register.
I20250901 01:53:48.191817 1071 heartbeater.cc:461] Registering TS with master...
I20250901 01:53:48.193550 2886 ts_manager.cc:194] Registered new tserver with Master: b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315)
I20250901 01:53:48.312466 2927 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:48.312825 2927 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:48.314991 2927 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:48.315807 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:48.317209 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:53:48.317457 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:53:48.317617 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:48.317963 2927 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:53:48.883553 2954 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:48.883747 2954 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:48.884521 2954 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:48.886370 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:48.886528 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:48.886694 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:48.886953 2954 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:53:49.933799 2963 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:49.934008 2963 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:49.934787 2963 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:49.937451 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:49.937655 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:49.937819 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:49.938074 2963 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:53:50.041687 416 master-stress-test.cc:577] Killing test thread 1/25
I20250901 01:53:51.589896 2976 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:51.590116 2976 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:51.590845 2976 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:51.593084 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:53:51.593289 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:53:51.593407 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:51.593637 2976 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:53:52.935273 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:53:52.935470 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:53:53.423875 2983 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:53.424099 2983 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:53.424736 2983 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:53.425300 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20250901 01:53:53.426494 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:53.426653 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:53.426766 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:53.427018 2983 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250901 01:53:55.186987 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 16: this message will repeat every 5th retry.
W20250901 01:53:55.324434 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 16: this message will repeat every 5th retry.
I20250901 01:53:57.070858 3006 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:57.071099 3006 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:57.071805 3006 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:57.074306 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:57.074681 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:53:57.074867 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:57.075153 3006 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:53:57.937076 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:53:57.937269 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:53:59.592226 3021 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:53:59.592465 3021 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:53:59.593125 3021 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:53:59.595081 2873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250901 01:53:59.595041 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:53:59.596582 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:53:59.596750 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:53:59.596892 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:53:59.597164 3021 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:54:02.939008 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:54:02.939244 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:54:04.555724 3051 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:54:04.555940 3051 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:54:04.556655 3051 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:54:04.559484 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:54:04.559885 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:54:04.560070 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:54:04.560417 3051 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250901 01:54:04.974221 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 21: this message will repeat every 5th retry.
W20250901 01:54:05.518493 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 21: this message will repeat every 5th retry.
I20250901 01:54:07.940840 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:54:07.941071 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:54:09.608754 3087 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:54:09.608951 3087 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:54:09.609593 3087 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:54:09.610630 2873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20250901 01:54:09.611953 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:54:09.612126 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:54:09.612277 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:54:09.612534 3087 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:54:11.289295 3103 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:54:11.289515 3103 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:54:11.290211 3103 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:54:11.292459 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:54:11.292673 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:54:11.292840 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:54:11.293126 3103 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:54:12.942793 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:54:12.943029 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:54:14.889206 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 26: this message will repeat every 5th retry.
W20250901 01:54:15.736976 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 26: this message will repeat every 5th retry.
W20250901 01:54:17.433729 1069 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:35805 (64 consecutive failures): Network error: Failed to ping master at 127.0.104.62:35805: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:54:17.622424 1213 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:36985 (176 consecutive failures): Network error: Failed to ping master at 127.0.104.60:36985: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:54:17.690578 1353 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:36985 (125 consecutive failures): Network error: Failed to ping master at 127.0.104.60:36985: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:54:17.944490 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:54:17.944703 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:54:22.345999 3168 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:54:22.346278 3168 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:54:22.346989 3168 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:54:22.347990 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250901 01:54:22.349370 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:54:22.349527 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:54:22.349699 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:54:22.349992 3168 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:54:22.946303 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:54:22.946564 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:54:24.642519 3183 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:54:24.642748 3183 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:54:24.643539 3183 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:54:24.645603 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:54:24.645787 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:54:24.645974 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:54:24.646306 3183 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250901 01:54:24.757449 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 31: this message will repeat every 5th retry.
W20250901 01:54:25.325369 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 31: this message will repeat every 5th retry.
I20250901 01:54:27.948199 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:54:27.948412 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:54:31.965396 3223 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:54:31.965638 3223 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:54:31.966490 3223 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:54:31.967689 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250901 01:54:31.969163 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:54:31.969496 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:54:31.969686 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:54:31.970026 3223 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:54:32.950511 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:54:32.950732 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:54:33.918449 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 36: this message will repeat every 5th retry.
W20250901 01:54:35.316393 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 36: this message will repeat every 5th retry.
I20250901 01:54:37.952335 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:54:37.952562 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:54:42.954238 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:54:42.954475 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:54:44.204026 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 41: this message will repeat every 5th retry.
W20250901 01:54:44.963495 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 41: this message will repeat every 5th retry.
I20250901 01:54:47.959493 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:54:47.959721 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:54:49.394186 3338 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:54:49.394420 3338 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:54:49.395308 3338 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:54:49.396381 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 01:54:49.398128 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:54:49.398391 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:54:49.398509 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:54:49.398778 3338 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:54:52.961427 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:54:52.961733 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:54:54.777490 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 46: this message will repeat every 5th retry.
W20250901 01:54:55.536885 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 46: this message will repeat every 5th retry.
I20250901 01:54:57.964426 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:54:57.964663 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:55:02.148223 3415 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:55:02.148494 3415 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:55:02.149312 3415 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:55:02.150768 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 01:55:02.152871 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:55:02.153177 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:55:02.153322 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:55:02.153661 3415 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:55:02.966459 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:55:02.966678 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:55:04.134701 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 51: this message will repeat every 5th retry.
W20250901 01:55:05.376479 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 51: this message will repeat every 5th retry.
I20250901 01:55:05.407696 3436 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:55:05.407919 3436 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:55:05.408672 3436 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:55:05.410981 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:55:05.411186 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:55:05.411324 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:55:05.411623 3436 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:55:06.589494 3448 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:55:06.589730 3448 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:55:06.590458 3448 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:55:06.592671 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:55:06.592864 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:55:06.593008 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:55:06.593299 3448 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:55:07.968256 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:55:07.968474 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:55:11.764290 3478 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:55:11.764550 3478 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:55:11.765252 3478 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:55:11.766836 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250901 01:55:11.768213 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:55:11.768424 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:55:11.768553 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:55:11.768836 3478 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:55:12.970387 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:55:12.970656 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:55:13.388033 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 56: this message will repeat every 5th retry.
W20250901 01:55:15.695478 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 56: this message will repeat every 5th retry.
W20250901 01:55:17.598984 1069 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:35805 (184 consecutive failures): Network error: Failed to ping master at 127.0.104.62:35805: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:55:17.793262 1212 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:35805 (281 consecutive failures): Network error: Failed to ping master at 127.0.104.62:35805: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:55:17.857184 1353 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:36985 (245 consecutive failures): Network error: Failed to ping master at 127.0.104.60:36985: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:55:17.972445 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:55:17.972694 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:55:22.974290 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:55:22.974532 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:55:23.043143 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 61: this message will repeat every 5th retry.
W20250901 01:55:26.045142 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 61: this message will repeat every 5th retry.
I20250901 01:55:27.976703 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:55:27.976944 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:55:29.829593 3575 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:55:29.829886 3575 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:55:29.830955 3575 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:55:29.832840 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 01:55:29.834803 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:55:29.835048 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:55:29.835171 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:55:29.835739 3575 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:55:32.086658 3594 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:55:32.086910 3594 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:55:32.087646 3594 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:55:32.089838 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:55:32.090065 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:55:32.090248 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:55:32.090533 3594 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:55:32.978551 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:55:32.978765 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:55:33.460561 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 66: this message will repeat every 5th retry.
W20250901 01:55:36.075057 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 66: this message will repeat every 5th retry.
I20250901 01:55:37.980944 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:55:37.981231 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:55:42.640923 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 71: this message will repeat every 5th retry.
I20250901 01:55:42.983100 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:55:42.983322 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:55:46.548774 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 71: this message will repeat every 5th retry.
I20250901 01:55:47.945664 2875 maintenance_manager.cc:419] P b557230fc3b142c4bf5f6f58d241905f: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033346
I20250901 01:55:47.986135 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:55:47.986404 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:55:48.005069 2874 maintenance_manager.cc:643] P b557230fc3b142c4bf5f6f58d241905f: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.058s user 0.034s sys 0.004s Metrics: {"bytes_written":22360,"cfile_init":1,"dirs.queue_time_us":507,"dirs.run_cpu_time_us":261,"dirs.run_wall_time_us":936,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":924,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":59,"thread_start_us":686,"threads_started":2,"wal-append.queue_time_us":470}
I20250901 01:55:48.006664 2875 maintenance_manager.cc:419] P b557230fc3b142c4bf5f6f58d241905f: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 3763 bytes on disk
I20250901 01:55:48.007711 2874 maintenance_manager.cc:643] P b557230fc3b142c4bf5f6f58d241905f: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20250901 01:55:48.883106 3688 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:55:48.883387 3688 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:55:48.884337 3688 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:55:48.885960 2873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250901 01:55:48.887567 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:55:48.887833 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:55:48.887957 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:55:48.888302 3688 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:55:52.988191 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:55:52.988443 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:55:53.510519 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 76: this message will repeat every 5th retry.
W20250901 01:55:56.880407 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 76: this message will repeat every 5th retry.
I20250901 01:55:57.990253 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:55:57.990583 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:56:02.992421 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:56:02.992666 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:56:03.111771 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 81: this message will repeat every 5th retry.
I20250901 01:56:06.207569 3832 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:56:06.207814 3832 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:56:06.208679 3832 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:56:06.209833 2873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 01:56:06.211475 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:56:06.211738 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:56:06.211872 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:56:06.212193 3832 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250901 01:56:06.561072 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 81: this message will repeat every 5th retry.
I20250901 01:56:07.994302 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:56:07.994522 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:56:12.996284 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:56:12.996490 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:56:13.089766 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 86: this message will repeat every 5th retry.
W20250901 01:56:16.751699 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 86: this message will repeat every 5th retry.
W20250901 01:56:17.757519 1069 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:35805 (304 consecutive failures): Network error: Failed to ping master at 127.0.104.62:35805: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:56:17.998238 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:56:17.998478 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:56:18.021102 1353 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:36985 (365 consecutive failures): Network error: Failed to ping master at 127.0.104.60:36985: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:56:18.165437 1212 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:35805 (459 consecutive failures): Network error: Failed to ping master at 127.0.104.62:35805: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:56:21.798032 3975 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:56:21.798372 3975 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:56:21.799672 3975 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:56:21.801422 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 01:56:21.803078 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:56:21.803632 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:56:21.803748 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:56:21.804056 3975 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:56:23.000406 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:56:23.000614 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:56:23.479578 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 91: this message will repeat every 5th retry.
W20250901 01:56:26.450131 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 91: this message will repeat every 5th retry.
I20250901 01:56:28.002561 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:56:28.002983 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:56:33.005153 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:56:33.005391 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:56:33.549747 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 96: this message will repeat every 5th retry.
I20250901 01:56:33.554713 4090 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:56:33.554963 4090 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:56:33.555765 4090 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:56:33.556779 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 01:56:33.558189 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:56:33.558423 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:56:33.558574 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:56:33.558866 4090 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250901 01:56:35.948731 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 96: this message will repeat every 5th retry.
I20250901 01:56:38.007593 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:56:38.007854 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:56:43.009512 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:56:43.009716 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:56:43.536566 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 101: this message will repeat every 5th retry.
I20250901 01:56:44.021066 4175 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:56:44.021314 4175 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:56:44.022042 4175 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:56:44.023224 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 01:56:44.024379 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:56:44.024622 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:56:44.024752 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:56:44.025040 4175 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250901 01:56:46.236212 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 101: this message will repeat every 5th retry.
I20250901 01:56:48.011337 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:56:48.011621 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:56:53.013556 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:56:53.013747 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:56:53.308097 4258 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:56:53.308348 4258 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:56:53.309083 4258 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:56:53.309739 2873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 01:56:53.311097 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:56:53.311384 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:56:53.311519 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:56:53.311779 4258 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250901 01:56:54.364734 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 106: this message will repeat every 5th retry.
W20250901 01:56:56.293506 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 106: this message will repeat every 5th retry.
I20250901 01:56:58.015277 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:56:58.015497 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:57:00.129029 4317 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:57:00.129282 4317 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:57:00.130124 4317 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:57:00.131335 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 01:57:00.132699 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:57:00.133077 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:57:00.133190 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:57:00.133481 4317 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:57:03.017225 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:57:03.017464 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:57:04.726327 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 111: this message will repeat every 5th retry.
W20250901 01:57:07.254846 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 111: this message will repeat every 5th retry.
I20250901 01:57:08.019148 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:57:08.019342 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:57:08.424424 4389 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:57:08.424657 4389 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:57:08.425417 4389 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:57:08.425915 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 01:57:08.427372 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:57:08.427559 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:57:08.427708 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:57:08.428012 4389 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:57:13.020977 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:57:13.021827 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:57:14.221237 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 116: this message will repeat every 5th retry.
W20250901 01:57:17.431245 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 116: this message will repeat every 5th retry.
W20250901 01:57:17.911917 1069 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:35805 (424 consecutive failures): Network error: Failed to ping master at 127.0.104.62:35805: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:57:18.023524 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:57:18.023743 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:57:18.179744 1353 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:36985 (485 consecutive failures): Network error: Failed to ping master at 127.0.104.60:36985: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:57:18.257244 1213 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:36985 (721 consecutive failures): Network error: Failed to ping master at 127.0.104.60:36985: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:57:19.753460 4490 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:57:19.753711 4490 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:57:19.754442 4490 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:57:19.755519 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 01:57:19.757424 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:57:19.757635 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:57:19.757745 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:57:19.757998 4490 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:57:23.025269 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:57:23.025478 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:57:24.624970 4537 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:57:24.625170 4537 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:57:24.625841 4537 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:57:24.627965 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:57:24.628178 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:57:24.628309 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:57:24.628588 4537 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250901 01:57:24.824730 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 121: this message will repeat every 5th retry.
W20250901 01:57:26.807013 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 121: this message will repeat every 5th retry.
I20250901 01:57:28.027545 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:57:28.027767 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:57:33.030078 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:57:33.030369 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:57:34.050624 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 126: this message will repeat every 5th retry.
I20250901 01:57:35.827910 4631 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:57:35.828145 4631 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:57:35.828826 4631 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:57:35.829813 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250901 01:57:35.831346 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:57:35.831516 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:57:35.831662 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:57:35.831908 4631 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250901 01:57:36.771842 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 126: this message will repeat every 5th retry.
I20250901 01:57:38.031847 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:57:38.032058 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:57:43.033799 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:57:43.034063 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:57:43.370015 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 131: this message will repeat every 5th retry.
W20250901 01:57:46.616606 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 131: this message will repeat every 5th retry.
I20250901 01:57:48.035646 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:57:48.035871 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:57:52.937794 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 136: this message will repeat every 5th retry.
I20250901 01:57:53.037329 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:57:53.037551 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:57:55.520088 4794 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:57:55.520305 4794 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:57:55.521277 4794 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:57:55.522274 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 01:57:55.523535 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:57:55.523723 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:57:55.523826 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:57:55.524096 4794 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250901 01:57:57.482108 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 136: this message will repeat every 5th retry.
I20250901 01:57:58.039446 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:57:58.039673 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:58:02.241509 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 141: this message will repeat every 5th retry.
I20250901 01:58:03.041437 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:58:03.041730 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:58:06.770769 4900 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:58:06.770952 4900 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:58:06.771588 4900 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:58:06.772631 2873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 01:58:06.773931 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:58:06.774183 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:58:06.774284 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:58:06.774541 4900 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250901 01:58:07.808353 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 141: this message will repeat every 5th retry.
I20250901 01:58:07.984846 4913 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:58:07.985065 4913 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:58:07.985752 4913 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:58:07.988092 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:58:07.988299 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:58:07.988406 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:58:07.988655 4913 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:58:08.043725 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:58:08.044049 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:58:11.614391 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 146: this message will repeat every 5th retry.
I20250901 01:58:13.045715 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:58:13.045917 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:58:17.517270 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 146: this message will repeat every 5th retry.
I20250901 01:58:18.047667 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:58:18.047878 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:58:18.068876 1069 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:35805 (544 consecutive failures): Network error: Failed to ping master at 127.0.104.62:35805: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:58:18.329360 1353 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:36985 (605 consecutive failures): Network error: Failed to ping master at 127.0.104.60:36985: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:58:18.344874 1212 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:35805 (828 consecutive failures): Network error: Failed to ping master at 127.0.104.62:35805: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:58:18.645723 5001 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:58:18.645929 5001 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:58:18.646688 5001 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:58:18.647775 2873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250901 01:58:18.649181 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:58:18.649344 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:58:18.649523 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:58:18.649781 5001 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250901 01:58:21.414832 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 151: this message will repeat every 5th retry.
I20250901 01:58:23.050043 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:58:23.050302 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:58:27.496260 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 151: this message will repeat every 5th retry.
I20250901 01:58:28.051822 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:58:28.052064 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:58:30.610730 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 156: this message will repeat every 5th retry.
I20250901 01:58:33.053475 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:58:33.053673 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:58:37.239871 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 156: this message will repeat every 5th retry.
I20250901 01:58:37.986552 5137 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:58:37.986784 5137 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:58:37.987684 5137 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:58:37.988776 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 01:58:37.989888 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
W20250901 01:58:37.990056 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
I20250901 01:58:37.990182 2870 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:58:37.990451 5137 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 01:58:38.055505 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:58:38.055756 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:58:39.668910 5158 raft_consensus.cc:491] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:58:39.669144 5158 raft_consensus.cc:513] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c35f5a36eeb4a93aecdd71e2d1c3a21" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35805 } } peers { permanent_uuid: "b557230fc3b142c4bf5f6f58d241905f" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 33025 } } peers { permanent_uuid: "835a442fbc4c48eb9148ca677e410a59" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 36985 } }
I20250901 01:58:39.669736 5158 leader_election.cc:290] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805), 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985)
W20250901 01:58:39.671729 2870 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2c35f5a36eeb4a93aecdd71e2d1c3a21 (127.0.104.62:35805): Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:58:39.671897 2873 leader_election.cc:336] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 835a442fbc4c48eb9148ca677e410a59 (127.0.104.60:36985): Network error: Client connection negotiation failed: client connection to 127.0.104.60:36985: connect: Connection refused (error 111)
I20250901 01:58:39.672044 2873 leader_election.cc:304] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b557230fc3b142c4bf5f6f58d241905f; no voters: 2c35f5a36eeb4a93aecdd71e2d1c3a21, 835a442fbc4c48eb9148ca677e410a59
I20250901 01:58:39.672300 5158 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b557230fc3b142c4bf5f6f58d241905f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250901 01:58:39.768095 1103 consensus_peers.cc:489] T 6b2f0235fadf41ed8e705af5059c4702 P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2f0235fadf41ed8e705af5059c4702. This is attempt 161: this message will repeat every 5th retry.
I20250901 01:58:41.043866 2875 maintenance_manager.cc:419] P b557230fc3b142c4bf5f6f58d241905f: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 3763 bytes on disk
I20250901 01:58:41.048903 2874 maintenance_manager.cc:643] P b557230fc3b142c4bf5f6f58d241905f: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.004s sys 0.000s Metrics: {"dirs.queue_time_us":411,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":113,"thread_start_us":538,"threads_started":2,"wal-append.queue_time_us":329}
I20250901 01:58:43.057317 2853 master_runner.cc:186] Error getting master registration for 127.0.104.62:35805: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111),
I20250901 01:58:43.057514 2853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 01:58:45.829063 1364 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:58:45.829358 1364 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:58:45.829471 1364 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
W20250901 01:58:45.829597 1364 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
F20250901 01:58:45.840459 1381 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1756691925 (unix time) try "date -d @1756691925" if you are using GNU date ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: T*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843441 1390 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843441 1390 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844631 1380 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843441 1390 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844631 1380 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844743 1377 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cbe4bb8385f54f7f9359f495d8c663a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843441 1390 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844631 1380 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844743 1377 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cbe4bb8385f54f7f9359f495d8c663a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844856 1371 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98a9d1400149426bab73e31aea263f5b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843441 1390 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844631 1380 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844743 1377 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cbe4bb8385f54f7f9359f495d8c663a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844856 1371 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98a9d1400149426bab73e31aea263f5b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845193 1385 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843441 1390 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844631 1380 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844743 1377 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cbe4bb8385f54f7f9359f495d8c663a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844856 1371 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98a9d1400149426bab73e31aea263f5b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845193 1385 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845403 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c1e876d5919f4c1f98a0ef9d1fcb9edc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843441 1390 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844631 1380 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844743 1377 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cbe4bb8385f54f7f9359f495d8c663a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844856 1371 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98a9d1400149426bab73e31aea263f5b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845193 1385 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845403 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c1e876d5919f4c1f98a0ef9d1fcb9edc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846478 1393 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843441 1390 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844631 1380 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844743 1377 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cbe4bb8385f54f7f9359f495d8c663a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844856 1371 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98a9d1400149426bab73e31aea263f5b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845193 1385 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845403 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c1e876d5919f4c1f98a0ef9d1fcb9edc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846478 1393 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846637 1382 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843441 1390 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844631 1380 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844743 1377 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cbe4bb8385f54f7f9359f495d8c663a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844856 1371 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98a9d1400149426bab73e31aea263f5b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845193 1385 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845403 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c1e876d5919f4c1f98a0ef9d1fcb9edc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846478 1393 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846637 1382 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846639 1378 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_820e14b2e7cf4bd7ad59ea83d8cb4d60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843441 1390 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844631 1380 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844743 1377 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cbe4bb8385f54f7f9359f495d8c663a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844856 1371 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98a9d1400149426bab73e31aea263f5b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845193 1385 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845403 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c1e876d5919f4c1f98a0ef9d1fcb9edc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846478 1393 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846637 1382 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846639 1378 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_820e14b2e7cf4bd7ad59ea83d8cb4d60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.847105 1389 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000001a0) received by PID 416 (TID 0x7ff0d3e4d700) from PID 416; stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843441 1390 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844631 1380 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844743 1377 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cbe4bb8385f54f7f9359f495d8c663a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844856 1371 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98a9d1400149426bab73e31aea263f5b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845193 1385 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845403 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c1e876d5919f4c1f98a0ef9d1fcb9edc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846478 1393 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846637 1382 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846639 1378 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_820e14b2e7cf4bd7ad59ea83d8cb4d60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.847105 1389 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.848428 1373 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ed3c253535084a3399b30083bb388f72 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843441 1390 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844631 1380 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844743 1377 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cbe4bb8385f54f7f9359f495d8c663a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844856 1371 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98a9d1400149426bab73e31aea263f5b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845193 1385 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845403 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c1e876d5919f4c1f98a0ef9d1fcb9edc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846478 1393 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846637 1382 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846639 1378 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_820e14b2e7cf4bd7ad59ea83d8cb4d60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.847105 1389 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.848428 1373 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ed3c253535084a3399b30083bb388f72 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.848558 1379 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f6c5cfd184ce474b883ce77bd225eb9f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843441 1390 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844631 1380 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844743 1377 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cbe4bb8385f54f7f9359f495d8c663a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844856 1371 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98a9d1400149426bab73e31aea263f5b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845193 1385 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845403 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c1e876d5919f4c1f98a0ef9d1fcb9edc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846478 1393 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846637 1382 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846639 1378 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_820e14b2e7cf4bd7ad59ea83d8cb4d60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.847105 1389 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.848428 1373 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ed3c253535084a3399b30083bb388f72 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.848558 1379 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f6c5cfd184ce474b883ce77bd225eb9f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.848618 1376 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2368f4dfae714931b490e2544bb27666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 01:58:45.841790 1375 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c7e1601c717640bba2d6ca6aa5478dbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842226 1387 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 01:58:45.842329 1388 master-stress-test.cc:Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111) Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842509 1396 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842557 1392 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842585 1383 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.842919 1386 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843007 1372 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0084f9e4b939410da1d13247891e6ac4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843236 1391 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843241 1384 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.843441 1390 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844631 1380 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844743 1377 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cbe4bb8385f54f7f9359f495d8c663a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.844856 1371 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98a9d1400149426bab73e31aea263f5b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845193 1385 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.845403 1370 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c1e876d5919f4c1f98a0ef9d1fcb9edc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846478 1393 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846637 1382 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.846639 1378 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_820e14b2e7cf4bd7ad59ea83d8cb4d60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.847105 1389 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.848428 1373 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ed3c253535084a3399b30083bb388f72 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.848558 1379 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f6c5cfd184ce474b883ce77bd225eb9f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.848618 1376 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2368f4dfae714931b490e2544bb27666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)F20250901 01:58:45.850086 1374 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_56fde480a00a4e62b84eedba22b0bade on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.62:35805,127.0.104.61:33025,127.0.104.60:36985, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.62:35805: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7ff0ebfa9980 (unknown) at ??:0
@ 0x7ff0e4f7dfb7 gsignal at ??:0
@ 0x7ff0e4f7f921 abort at ??:0
@ 0x7ff0e7cb5dcd google::LogMessage::Fail() at ??:0
@ 0x7ff0e7cb9b93 google::LogMessage::SendToLog() at ??:0
@ 0x7ff0e7cb57cc google::LogMessage::Flush() at ??:0
@ 0x7ff0e7cb6f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40c481 kudu::MasterStressTest::AlterTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:328
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7ff0e56056df (unknown) at ??:0
@ 0x7ff0ebf9e6db start_thread at ??:0
@ 0x7ff0e506071f clone at ??:0
W20250901 01:58:45.997699 1103 consensus_peers.cc:489] T 4e1a5e7ec3f842ffa6f24e5b315eb6dd P e3c774a4edf44d6098a2332e734967a5 -> Peer b65b3154aefd49f7a37f70e9207e9853 (127.0.104.1:45315): Couldn't send request to peer b65b3154aefd49f7a37f70e9207e9853. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e1a5e7ec3f842ffa6f24e5b315eb6dd. This is attempt 161: this message will repeat every 5th retry.