Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251212 02:55:03.691951 12359 test_util.cc:276] Using random seed: 1667128963
2025-12-12T02:55:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-12T02:55:03Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskaeoKhp/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskaeoKhp/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/security.properties
2025-12-12 02:55:09: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskaeoKhp/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskaeoKhp/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20251212 02:55:20.109830 12359 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
/tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.254:40649
--webserver_interface=127.12.17.254
--webserver_port=0
--builtin_ntp_servers=127.12.17.212:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251212 02:55:20.435137 12602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251212 02:55:20.435516 12602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251212 02:55:20.435657 12602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251212 02:55:20.443929 12602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251212 02:55:20.444046 12602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251212 02:55:20.444131 12602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251212 02:55:20.444207 12602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251212 02:55:20.454126 12602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.17.212:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.254:40649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.17.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 12 Dec 2025 02:43:20 UTC on 24a791456cd2
build id 9472
ASAN enabled
I20251212 02:55:20.455818 12602 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 02:55:20.457659 12602 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 02:55:20.466120 12607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:20.466146 12610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:20.466094 12608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:20.467322 12602 server_base.cc:1047] running on GCE node
I20251212 02:55:20.468058 12602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251212 02:55:20.469594 12602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251212 02:55:20.470815 12602 hybrid_clock.cc:648] HybridClock initialized: now 1765508120470774 us; error 30 us; skew 500 ppm
I20251212 02:55:20.471194 12602 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251212 02:55:20.473730 12602 webserver.cc:492] Webserver started at http://127.12.17.254:35303/ using document root <none> and password file <none>
I20251212 02:55:20.474515 12602 fs_manager.cc:362] Metadata directory not provided
I20251212 02:55:20.474623 12602 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 02:55:20.474977 12602 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 02:55:20.477475 12602 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/instance:
uuid: "62ed31a99dac451981f4562901f11e87"
format_stamp: "Formatted at 2025-12-12 02:55:20 on dist-test-slave-8cz7"
I20251212 02:55:20.478281 12602 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal/instance:
uuid: "62ed31a99dac451981f4562901f11e87"
format_stamp: "Formatted at 2025-12-12 02:55:20 on dist-test-slave-8cz7"
I20251212 02:55:20.483876 12602 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251212 02:55:20.487668 12616 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:20.489260 12602 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251212 02:55:20.489441 12602 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
uuid: "62ed31a99dac451981f4562901f11e87"
format_stamp: "Formatted at 2025-12-12 02:55:20 on dist-test-slave-8cz7"
I20251212 02:55:20.489655 12602 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 02:55:20.515010 12602 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 02:55:20.516023 12602 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 02:55:20.516330 12602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 02:55:20.538318 12602 rpc_server.cc:307] RPC server started. Bound to: 127.12.17.254:40649
I20251212 02:55:20.538338 12667 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.17.254:40649 every 8 connection(s)
I20251212 02:55:20.540007 12602 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/info.pb
I20251212 02:55:20.544426 12668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:20.548368 12359 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu as pid 12602
I20251212 02:55:20.548717 12359 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal/instance
I20251212 02:55:20.552464 12359 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
/tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.253:46623
--webserver_interface=127.12.17.253
--webserver_port=0
--builtin_ntp_servers=127.12.17.212:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251212 02:55:20.553062 12668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } has no permanent_uuid. Determining permanent_uuid...
I20251212 02:55:20.574568 12668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } has no permanent_uuid. Determining permanent_uuid...
W20251212 02:55:20.575594 12617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:55:20.577056 12668 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:20.626436 12668 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } attempt: 1
W20251212 02:55:20.628257 12668 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:20.696538 12668 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } attempt: 2
W20251212 02:55:20.698454 12668 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:20.789752 12668 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } attempt: 3
W20251212 02:55:20.791618 12668 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:55:20.885793 12669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251212 02:55:20.886140 12669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251212 02:55:20.886229 12669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251212 02:55:20.894711 12669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251212 02:55:20.894809 12669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251212 02:55:20.894892 12669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251212 02:55:20.894948 12669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251212 02:55:20.904825 12669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.17.212:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.253:46623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.17.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 12 Dec 2025 02:43:20 UTC on 24a791456cd2
build id 9472
ASAN enabled
I20251212 02:55:20.906471 12669 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 02:55:20.908226 12669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 02:55:20.916285 12678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:20.916291 12677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:20.916913 12680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:20.917433 12669 server_base.cc:1047] running on GCE node
I20251212 02:55:20.918135 12669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251212 02:55:20.919507 12669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251212 02:55:20.920733 12669 hybrid_clock.cc:648] HybridClock initialized: now 1765508120920687 us; error 43 us; skew 500 ppm
I20251212 02:55:20.921065 12669 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251212 02:55:20.923221 12669 webserver.cc:492] Webserver started at http://127.12.17.253:36499/ using document root <none> and password file <none>
I20251212 02:55:20.923964 12669 fs_manager.cc:362] Metadata directory not provided
I20251212 02:55:20.924068 12669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 02:55:20.924424 12669 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 02:55:20.926775 12669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data/instance:
uuid: "ad856a8b02154355b5da573e8fffcb16"
format_stamp: "Formatted at 2025-12-12 02:55:20 on dist-test-slave-8cz7"
I20251212 02:55:20.927534 12669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal/instance:
uuid: "ad856a8b02154355b5da573e8fffcb16"
format_stamp: "Formatted at 2025-12-12 02:55:20 on dist-test-slave-8cz7"
I20251212 02:55:20.932808 12669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251212 02:55:20.934895 12668 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } attempt: 4
I20251212 02:55:20.936467 12686 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20251212 02:55:20.936813 12668 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:20.938040 12669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251212 02:55:20.938202 12669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data,/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal
uuid: "ad856a8b02154355b5da573e8fffcb16"
format_stamp: "Formatted at 2025-12-12 02:55:20 on dist-test-slave-8cz7"
I20251212 02:55:20.938405 12669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 02:55:20.957074 12669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 02:55:20.957988 12669 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 02:55:20.958216 12669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 02:55:20.977927 12669 rpc_server.cc:307] RPC server started. Bound to: 127.12.17.253:46623
I20251212 02:55:20.977947 12737 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.17.253:46623 every 8 connection(s)
I20251212 02:55:20.979439 12669 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data/info.pb
I20251212 02:55:20.980791 12359 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu as pid 12669
I20251212 02:55:20.980965 12359 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal/instance
I20251212 02:55:20.982065 12359 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
/tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.252:44417
--webserver_interface=127.12.17.252
--webserver_port=0
--builtin_ntp_servers=127.12.17.212:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251212 02:55:20.983767 12738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:20.989908 12738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } has no permanent_uuid. Determining permanent_uuid...
I20251212 02:55:21.003152 12738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } has no permanent_uuid. Determining permanent_uuid...
I20251212 02:55:21.012959 12738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } has no permanent_uuid. Determining permanent_uuid...
W20251212 02:55:21.014257 12688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:55:21.015889 12738 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.12.17.252:44417: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:21.065183 12738 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } attempt: 1
W20251212 02:55:21.067085 12738 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.12.17.252:44417: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:21.135335 12738 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } attempt: 2
W20251212 02:55:21.137447 12738 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.12.17.252:44417: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:21.228773 12738 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } attempt: 3
W20251212 02:55:21.230865 12738 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.12.17.252:44417: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:21.236063 12668 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } attempt: 5
I20251212 02:55:21.242004 12668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } has no permanent_uuid. Determining permanent_uuid...
W20251212 02:55:21.243748 12668 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.12.17.252:44417: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:21.294950 12668 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } attempt: 1
W20251212 02:55:21.296800 12668 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.12.17.252:44417: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:55:21.318504 12739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251212 02:55:21.318842 12739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251212 02:55:21.318972 12739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251212 02:55:21.327356 12739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251212 02:55:21.327471 12739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251212 02:55:21.327560 12739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251212 02:55:21.327625 12739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251212 02:55:21.337399 12739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.17.212:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.252:44417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.17.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 12 Dec 2025 02:43:20 UTC on 24a791456cd2
build id 9472
ASAN enabled
I20251212 02:55:21.339039 12739 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 02:55:21.340826 12739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 02:55:21.349555 12747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:21.349593 12748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:21.350093 12750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:21.351114 12739 server_base.cc:1047] running on GCE node
I20251212 02:55:21.351788 12739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251212 02:55:21.353140 12739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251212 02:55:21.354382 12739 hybrid_clock.cc:648] HybridClock initialized: now 1765508121354346 us; error 32 us; skew 500 ppm
I20251212 02:55:21.354714 12739 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251212 02:55:21.356894 12739 webserver.cc:492] Webserver started at http://127.12.17.252:42635/ using document root <none> and password file <none>
I20251212 02:55:21.357678 12739 fs_manager.cc:362] Metadata directory not provided
I20251212 02:55:21.357784 12739 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 02:55:21.358132 12739 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 02:55:21.360437 12739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/instance:
uuid: "7823e6e682ac46a09f968e0742506d6e"
format_stamp: "Formatted at 2025-12-12 02:55:21 on dist-test-slave-8cz7"
I20251212 02:55:21.361186 12739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal/instance:
uuid: "7823e6e682ac46a09f968e0742506d6e"
format_stamp: "Formatted at 2025-12-12 02:55:21 on dist-test-slave-8cz7"
I20251212 02:55:21.365062 12668 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } attempt: 2
I20251212 02:55:21.366776 12739 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
W20251212 02:55:21.366982 12668 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.12.17.252:44417: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:21.370187 12756 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:21.371562 12739 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251212 02:55:21.371721 12739 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data,/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
uuid: "7823e6e682ac46a09f968e0742506d6e"
format_stamp: "Formatted at 2025-12-12 02:55:21 on dist-test-slave-8cz7"
I20251212 02:55:21.371922 12739 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 02:55:21.374101 12738 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } attempt: 4
W20251212 02:55:21.375949 12738 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.12.17.252:44417: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:21.394397 12739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 02:55:21.395391 12739 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 02:55:21.395659 12739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 02:55:21.416419 12739 rpc_server.cc:307] RPC server started. Bound to: 127.12.17.252:44417
I20251212 02:55:21.416451 12807 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.17.252:44417 every 8 connection(s)
I20251212 02:55:21.418025 12739 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/info.pb
I20251212 02:55:21.418624 12359 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu as pid 12739
I20251212 02:55:21.418825 12359 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal/instance
I20251212 02:55:21.422389 12808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:21.429579 12808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } has no permanent_uuid. Determining permanent_uuid...
I20251212 02:55:21.443317 12808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } has no permanent_uuid. Determining permanent_uuid...
I20251212 02:55:21.456406 12808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } has no permanent_uuid. Determining permanent_uuid...
I20251212 02:55:21.466463 12808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Bootstrap starting.
I20251212 02:55:21.469607 12808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:21.470832 12808 log.cc:826] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Log is configured to *not* fsync() on all Append() calls
I20251212 02:55:21.473223 12668 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } attempt: 3
I20251212 02:55:21.473295 12808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: No bootstrap required, opened a new log
I20251212 02:55:21.479930 12808 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:21.480255 12808 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:21.480372 12808 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7823e6e682ac46a09f968e0742506d6e, State: Initialized, Role: FOLLOWER
I20251212 02:55:21.480962 12808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:21.481978 12815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:21.482241 12815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:21.483043 12808 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: configured and running, proceeding with master startup.
I20251212 02:55:21.483655 12668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Bootstrap starting.
I20251212 02:55:21.486922 12668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:21.488243 12668 log.cc:826] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Log is configured to *not* fsync() on all Append() calls
I20251212 02:55:21.492074 12668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: No bootstrap required, opened a new log
I20251212 02:55:21.493743 12739 master_runner.cc:186] Error getting master registration for 127.12.17.254:40649: OK, instance_id { permanent_uuid: "62ed31a99dac451981f4562901f11e87" instance_seqno: 1765508120532406 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251212 02:55:21.499778 12668 raft_consensus.cc:359] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:21.500157 12668 raft_consensus.cc:385] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:21.500299 12668 raft_consensus.cc:740] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ed31a99dac451981f4562901f11e87, State: Initialized, Role: FOLLOWER
I20251212 02:55:21.500340 12739 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:55:21.501333 12668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
W20251212 02:55:21.504802 12830 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251212 02:55:21.505038 12830 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251212 02:55:21.505805 12833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:21.506208 12833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:21.507381 12668 sys_catalog.cc:565] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: configured and running, proceeding with master startup.
W20251212 02:55:21.519197 12622 tablet.cc:2378] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251212 02:55:21.530725 12602 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: OK, instance_id { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" instance_seqno: 1765508120972568 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251212 02:55:21.547698 12602 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251212 02:55:21.541458 12848 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251212 02:55:21.548007 12848 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251212 02:55:21.596326 12833 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:21.596660 12833 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:21.598650 12833 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
I20251212 02:55:21.600431 12783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ed31a99dac451981f4562901f11e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7823e6e682ac46a09f968e0742506d6e" is_pre_election: true
I20251212 02:55:21.601042 12783 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62ed31a99dac451981f4562901f11e87 in term 0.
I20251212 02:55:21.601822 12618 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e; no voters:
I20251212 02:55:21.602337 12833 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251212 02:55:21.602496 12833 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 02:55:21.602591 12833 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:21.605808 12833 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:21.607735 12783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ed31a99dac451981f4562901f11e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7823e6e682ac46a09f968e0742506d6e"
I20251212 02:55:21.607952 12783 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:21.611351 12783 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62ed31a99dac451981f4562901f11e87 in term 1.
I20251212 02:55:21.612613 12833 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 1 election: Requested vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
I20251212 02:55:21.613328 12618 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e; no voters:
I20251212 02:55:21.613739 12833 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 1 FOLLOWER]: Leader election won for term 1
I20251212 02:55:21.629648 12833 raft_consensus.cc:697] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 1 LEADER]: Becoming Leader. State: Replica: 62ed31a99dac451981f4562901f11e87, State: Running, Role: LEADER
I20251212 02:55:21.630415 12833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
W20251212 02:55:21.647790 12762 tablet.cc:2378] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251212 02:55:21.671149 12850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ed31a99dac451981f4562901f11e87. Latest consensus state: current_term: 1 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:21.671558 12850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: LEADER
I20251212 02:55:21.675155 12738 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } attempt: 5
I20251212 02:55:21.688593 12738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Bootstrap starting.
I20251212 02:55:21.689503 12852 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251212 02:55:21.691830 12852 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251212 02:55:21.692880 12738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:21.694283 12738 log.cc:826] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Log is configured to *not* fsync() on all Append() calls
I20251212 02:55:21.703796 12738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: No bootstrap required, opened a new log
I20251212 02:55:21.704434 12783 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 1 FOLLOWER]: Refusing update from remote peer 62ed31a99dac451981f4562901f11e87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251212 02:55:21.705717 12850 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 02:55:21.715582 12738 raft_consensus.cc:359] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:21.728327 12738 raft_consensus.cc:385] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:21.728597 12738 raft_consensus.cc:740] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad856a8b02154355b5da573e8fffcb16, State: Initialized, Role: FOLLOWER
I20251212 02:55:21.729800 12738 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:21.732585 12857 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:21.741348 12857 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:21.735397 12711 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:21.735558 12713 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ed31a99dac451981f4562901f11e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad856a8b02154355b5da573e8fffcb16" is_pre_election: true
I20251212 02:55:21.735802 12712 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ed31a99dac451981f4562901f11e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad856a8b02154355b5da573e8fffcb16"
I20251212 02:55:21.745793 12711 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 1 FOLLOWER]: Refusing update from remote peer 62ed31a99dac451981f4562901f11e87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251212 02:55:21.735013 12738 sys_catalog.cc:565] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: configured and running, proceeding with master startup.
I20251212 02:55:21.746802 12850 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 02:55:21.755254 12815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ed31a99dac451981f4562901f11e87. Latest consensus state: current_term: 1 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:21.755579 12815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:21.759122 12854 mvcc.cc:204] Tried to move back new op lower bound from 7231521266484342784 to 7231521266204733440. Current Snapshot: MvccSnapshot[applied={T|T < 7231521266484342784}]
I20251212 02:55:21.761627 12850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:21.761868 12850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: LEADER
I20251212 02:55:21.766685 12815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:21.766908 12815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:21.770151 12852 catalog_manager.cc:1357] Generated new cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:21.770287 12852 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251212 02:55:21.772269 12858 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ed31a99dac451981f4562901f11e87. Latest consensus state: current_term: 1 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:21.772543 12858 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:21.773708 12833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:21.773972 12833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: LEADER
I20251212 02:55:21.786067 12860 mvcc.cc:204] Tried to move back new op lower bound from 7231521266484342784 to 7231521266204733440. Current Snapshot: MvccSnapshot[applied={T|T < 7231521266484342784}]
I20251212 02:55:21.786994 12858 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:21.787228 12858 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:21.812079 12852 catalog_manager.cc:1380] Generated new certificate authority record
I20251212 02:55:21.814008 12852 catalog_manager.cc:1514] Loading token signing keys...
I20251212 02:55:21.847391 12852 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Generated new TSK 0
I20251212 02:55:21.848752 12852 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251212 02:55:21.850478 12852 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251212 02:55:21.851840 12852 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251212 02:55:21.861231 12359 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
/tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.17.193:0
--local_ip_for_outbound_sockets=127.12.17.193
--webserver_interface=127.12.17.193
--webserver_port=0
--tserver_master_addrs=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--builtin_ntp_servers=127.12.17.212:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251212 02:55:21.877810 12879 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20251212 02:55:22.313548 12878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251212 02:55:22.313871 12878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251212 02:55:22.313966 12878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251212 02:55:22.322438 12878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251212 02:55:22.322610 12878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.17.193
I20251212 02:55:22.332614 12878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.17.212:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.17.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.17.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.17.193
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 12 Dec 2025 02:43:20 UTC on 24a791456cd2
build id 9472
ASAN enabled
I20251212 02:55:22.334245 12878 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 02:55:22.335932 12878 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 02:55:22.345563 12885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:22.345932 12884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:22.350224 12889 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62ed31a99dac451981f4562901f11e87)
I20251212 02:55:22.350481 12889 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
W20251212 02:55:22.351022 12887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:22.351764 12889 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
I20251212 02:55:22.352435 12783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad856a8b02154355b5da573e8fffcb16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7823e6e682ac46a09f968e0742506d6e" is_pre_election: true
I20251212 02:55:22.352845 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad856a8b02154355b5da573e8fffcb16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "62ed31a99dac451981f4562901f11e87" is_pre_election: true
I20251212 02:55:22.353616 12687 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:22.353780 12878 server_base.cc:1047] running on GCE node
I20251212 02:55:22.354274 12889 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251212 02:55:22.354539 12878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251212 02:55:22.355614 12878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251212 02:55:22.356845 12878 hybrid_clock.cc:648] HybridClock initialized: now 1765508122356798 us; error 41 us; skew 500 ppm
I20251212 02:55:22.357234 12878 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251212 02:55:22.359537 12878 webserver.cc:492] Webserver started at http://127.12.17.193:43795/ using document root <none> and password file <none>
I20251212 02:55:22.360289 12878 fs_manager.cc:362] Metadata directory not provided
I20251212 02:55:22.360391 12878 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 02:55:22.360754 12878 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 02:55:22.363175 12878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/data/instance:
uuid: "480bcf5c1f394464b302900a89ca5bdf"
format_stamp: "Formatted at 2025-12-12 02:55:22 on dist-test-slave-8cz7"
I20251212 02:55:22.364002 12878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/wal/instance:
uuid: "480bcf5c1f394464b302900a89ca5bdf"
format_stamp: "Formatted at 2025-12-12 02:55:22 on dist-test-slave-8cz7"
I20251212 02:55:22.369817 12878 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20251212 02:55:22.373436 12894 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:22.374879 12878 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251212 02:55:22.375126 12878 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/wal
uuid: "480bcf5c1f394464b302900a89ca5bdf"
format_stamp: "Formatted at 2025-12-12 02:55:22 on dist-test-slave-8cz7"
I20251212 02:55:22.375368 12878 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 02:55:22.395478 12878 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 02:55:22.396451 12878 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 02:55:22.396749 12878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 02:55:22.397971 12878 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 02:55:22.399801 12878 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251212 02:55:22.399914 12878 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:22.400050 12878 ts_tablet_manager.cc:616] Registered 0 tablets
I20251212 02:55:22.400136 12878 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:22.435613 12878 rpc_server.cc:307] RPC server started. Bound to: 127.12.17.193:40431
I20251212 02:55:22.435645 13006 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.17.193:40431 every 8 connection(s)
I20251212 02:55:22.437637 12878 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/data/info.pb
I20251212 02:55:22.441660 12359 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu as pid 12878
I20251212 02:55:22.441892 12359 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/wal/instance
I20251212 02:55:22.448498 12359 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
/tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.17.194:0
--local_ip_for_outbound_sockets=127.12.17.194
--webserver_interface=127.12.17.194
--webserver_port=0
--tserver_master_addrs=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--builtin_ntp_servers=127.12.17.212:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251212 02:55:22.468920 13009 heartbeater.cc:344] Connected to a master server at 127.12.17.253:46623
I20251212 02:55:22.469031 13008 heartbeater.cc:344] Connected to a master server at 127.12.17.254:40649
I20251212 02:55:22.469107 13007 heartbeater.cc:344] Connected to a master server at 127.12.17.252:44417
I20251212 02:55:22.469484 13008 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:22.469483 13007 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:22.469487 13009 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:22.471047 13009 heartbeater.cc:507] Master 127.12.17.253:46623 requested a full tablet report, sending...
I20251212 02:55:22.471084 13008 heartbeater.cc:507] Master 127.12.17.254:40649 requested a full tablet report, sending...
I20251212 02:55:22.471240 13007 heartbeater.cc:507] Master 127.12.17.252:44417 requested a full tablet report, sending...
I20251212 02:55:22.473882 12703 ts_manager.cc:194] Registered new tserver with Master: 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431)
I20251212 02:55:22.473912 12773 ts_manager.cc:194] Registered new tserver with Master: 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431)
I20251212 02:55:22.475257 12633 ts_manager.cc:194] Registered new tserver with Master: 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431)
I20251212 02:55:22.476776 12633 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.12.17.193:47679
I20251212 02:55:22.507126 12830 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:22.507248 12830 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: loading cluster ID for follower catalog manager: success
I20251212 02:55:22.510088 12830 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: acquiring CA information for follower catalog manager: success
I20251212 02:55:22.511847 12830 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251212 02:55:22.828469 13021 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62ed31a99dac451981f4562901f11e87)
I20251212 02:55:22.828677 13021 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:22.830024 13021 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
I20251212 02:55:22.830790 12712 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7823e6e682ac46a09f968e0742506d6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ad856a8b02154355b5da573e8fffcb16" is_pre_election: true
I20251212 02:55:22.830821 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7823e6e682ac46a09f968e0742506d6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "62ed31a99dac451981f4562901f11e87" is_pre_election: true
I20251212 02:55:22.831542 12757 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e; no voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:22.832001 13021 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251212 02:55:22.836911 13017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251212 02:55:22.837220 13017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251212 02:55:22.837349 13017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251212 02:55:22.845820 13017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251212 02:55:22.846010 13017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.17.194
I20251212 02:55:22.856366 13017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.17.212:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.17.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.17.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.17.194
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 12 Dec 2025 02:43:20 UTC on 24a791456cd2
build id 9472
ASAN enabled
I20251212 02:55:22.858119 13017 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 02:55:22.859917 13017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251212 02:55:22.868122 13026 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62ed31a99dac451981f4562901f11e87)
I20251212 02:55:22.868801 13026 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:22.870178 13026 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
I20251212 02:55:22.870316 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad856a8b02154355b5da573e8fffcb16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "62ed31a99dac451981f4562901f11e87" is_pre_election: true
I20251212 02:55:22.870721 12783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad856a8b02154355b5da573e8fffcb16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7823e6e682ac46a09f968e0742506d6e" is_pre_election: true
I20251212 02:55:22.870986 12783 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad856a8b02154355b5da573e8fffcb16 in term 1.
I20251212 02:55:22.871527 12688 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87
W20251212 02:55:22.871500 13023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:22.872059 13026 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251212 02:55:22.872210 13026 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62ed31a99dac451981f4562901f11e87)
W20251212 02:55:22.872244 13027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:22.872320 13026 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 1 FOLLOWER]: Advancing to term 2
W20251212 02:55:22.872398 13024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:22.874348 13017 server_base.cc:1047] running on GCE node
I20251212 02:55:22.874962 13017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251212 02:55:22.875034 13026 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:22.875991 13026 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 2 election: Requested vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
I20251212 02:55:22.876112 12643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad856a8b02154355b5da573e8fffcb16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "62ed31a99dac451981f4562901f11e87"
I20251212 02:55:22.876104 12783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad856a8b02154355b5da573e8fffcb16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7823e6e682ac46a09f968e0742506d6e"
I20251212 02:55:22.876333 12783 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 1 FOLLOWER]: Advancing to term 2
I20251212 02:55:22.876578 13017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251212 02:55:22.877815 13017 hybrid_clock.cc:648] HybridClock initialized: now 1765508122877722 us; error 75 us; skew 500 ppm
I20251212 02:55:22.878206 13017 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251212 02:55:22.878917 12783 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad856a8b02154355b5da573e8fffcb16 in term 2.
I20251212 02:55:22.879370 12688 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87
I20251212 02:55:22.879707 13026 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 2 FOLLOWER]: Leader election won for term 2
I20251212 02:55:22.884136 13017 webserver.cc:492] Webserver started at http://127.12.17.194:42961/ using document root <none> and password file <none>
I20251212 02:55:22.884357 13026 raft_consensus.cc:697] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 2 LEADER]: Becoming Leader. State: Replica: ad856a8b02154355b5da573e8fffcb16, State: Running, Role: LEADER
I20251212 02:55:22.885020 13026 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:22.885257 13017 fs_manager.cc:362] Metadata directory not provided
I20251212 02:55:22.885388 13017 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 02:55:22.885820 13017 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 02:55:22.889389 13017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/data/instance:
uuid: "8240d2efd35f443a9515f0db38f09f75"
format_stamp: "Formatted at 2025-12-12 02:55:22 on dist-test-slave-8cz7"
I20251212 02:55:22.890661 13017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/wal/instance:
uuid: "8240d2efd35f443a9515f0db38f09f75"
format_stamp: "Formatted at 2025-12-12 02:55:22 on dist-test-slave-8cz7"
I20251212 02:55:22.892840 12879 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:22.892978 12879 catalog_manager.cc:1562] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: loading cluster ID for follower catalog manager: success
I20251212 02:55:22.897791 12879 catalog_manager.cc:1584] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: acquiring CA information for follower catalog manager: success
I20251212 02:55:22.901235 12879 catalog_manager.cc:1612] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251212 02:55:22.902029 13031 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad856a8b02154355b5da573e8fffcb16. Latest consensus state: current_term: 2 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:22.902429 13031 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: LEADER
I20251212 02:55:22.903432 13037 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251212 02:55:22.906199 13037 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251212 02:55:22.906404 12712 raft_consensus.cc:1240] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 2 LEADER]: Rejecting Update request from peer 62ed31a99dac451981f4562901f11e87 for earlier term 1. Current term is 2. Ops: []
I20251212 02:55:22.907369 13037 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:22.907294 13035 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251212 02:55:22.907593 13037 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251212 02:55:22.909801 13037 catalog_manager.cc:1514] Loading token signing keys...
I20251212 02:55:22.911307 13037 catalog_manager.cc:6038] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Loaded TSK: 0
I20251212 02:55:22.912745 13037 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251212 02:55:22.913147 13017 fs_manager.cc:696] Time spent creating directory manager: real 0.022s user 0.009s sys 0.008s
I20251212 02:55:22.914408 13037 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251212 02:55:22.915967 13037 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251212 02:55:22.917528 13035 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [LEADER]: Peer ad856a8b02154355b5da573e8fffcb16 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20251212 02:55:22.918617 13039 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 1 LEADER]: Stepping down as leader of term 1
I20251212 02:55:22.918730 13039 raft_consensus.cc:740] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62ed31a99dac451981f4562901f11e87, State: Running, Role: LEADER
I20251212 02:55:22.919077 12783 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 2 FOLLOWER]: Rejecting Update request from peer 62ed31a99dac451981f4562901f11e87 for earlier term 1. Current term is 2. Ops: []
I20251212 02:55:22.919076 13039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:22.919569 13039 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 1 FOLLOWER]: Advancing to term 2
I20251212 02:55:22.925799 13040 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:22.927601 13017 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.010s sys 0.000s
I20251212 02:55:22.927783 13017 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/wal
uuid: "8240d2efd35f443a9515f0db38f09f75"
format_stamp: "Formatted at 2025-12-12 02:55:22 on dist-test-slave-8cz7"
I20251212 02:55:22.928035 13017 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 02:55:22.947873 13017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 02:55:22.949018 13017 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 02:55:22.949384 13017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 02:55:22.950590 13017 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 02:55:22.952771 13017 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251212 02:55:22.952888 13017 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:22.952997 13017 ts_tablet_manager.cc:616] Registered 0 tablets
I20251212 02:55:22.953080 13017 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:22.981341 13008 heartbeater.cc:499] Master 127.12.17.254:40649 was elected leader, sending a full tablet report...
I20251212 02:55:22.999272 13017 rpc_server.cc:307] RPC server started. Bound to: 127.12.17.194:43037
I20251212 02:55:22.999328 13152 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.17.194:43037 every 8 connection(s)
I20251212 02:55:23.001107 13017 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/data/info.pb
I20251212 02:55:23.009749 12359 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu as pid 13017
I20251212 02:55:23.010018 12359 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/wal/instance
I20251212 02:55:23.017172 12359 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
/tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.17.195:0
--local_ip_for_outbound_sockets=127.12.17.195
--webserver_interface=127.12.17.195
--webserver_port=0
--tserver_master_addrs=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--builtin_ntp_servers=127.12.17.212:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251212 02:55:23.033614 13154 heartbeater.cc:344] Connected to a master server at 127.12.17.254:40649
I20251212 02:55:23.034046 13154 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:23.035247 13154 heartbeater.cc:507] Master 127.12.17.254:40649 requested a full tablet report, sending...
I20251212 02:55:23.037019 13153 heartbeater.cc:344] Connected to a master server at 127.12.17.252:44417
I20251212 02:55:23.037158 13153 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:23.037607 12633 ts_manager.cc:194] Registered new tserver with Master: 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
I20251212 02:55:23.037719 13153 heartbeater.cc:507] Master 127.12.17.252:44417 requested a full tablet report, sending...
I20251212 02:55:23.038475 13155 heartbeater.cc:344] Connected to a master server at 127.12.17.253:46623
I20251212 02:55:23.038589 13155 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:23.038882 12773 ts_manager.cc:194] Registered new tserver with Master: 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
I20251212 02:55:23.039108 13155 heartbeater.cc:507] Master 127.12.17.253:46623 requested a full tablet report, sending...
I20251212 02:55:23.040071 12703 ts_manager.cc:194] Registered new tserver with Master: 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
I20251212 02:55:23.041422 12703 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.12.17.194:58003
W20251212 02:55:23.361974 13162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251212 02:55:23.362273 13162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251212 02:55:23.362404 13162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251212 02:55:23.370100 12643 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 2 FOLLOWER]: Refusing update from remote peer ad856a8b02154355b5da573e8fffcb16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251212 02:55:23.370898 13031 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20251212 02:55:23.371658 13162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251212 02:55:23.371858 13162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.17.195
I20251212 02:55:23.372968 13039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad856a8b02154355b5da573e8fffcb16. Latest consensus state: current_term: 2 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:23.373266 13039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:23.375043 13026 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:23.375293 13026 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: LEADER
I20251212 02:55:23.375744 12783 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 2 FOLLOWER]: Refusing update from remote peer ad856a8b02154355b5da573e8fffcb16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251212 02:55:23.376439 13031 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251212 02:55:23.377007 13039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:23.377230 13039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:23.379112 13173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:23.379455 13173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:23.379388 13170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad856a8b02154355b5da573e8fffcb16. Latest consensus state: current_term: 2 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:23.379765 13170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:23.384508 13162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.17.212:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.17.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.17.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.17.195
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 12 Dec 2025 02:43:20 UTC on 24a791456cd2
build id 9472
ASAN enabled
I20251212 02:55:23.386264 13162 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 02:55:23.388087 13162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 02:55:23.398075 13176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:23.398219 13175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:23.399036 13178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:23.399461 13162 server_base.cc:1047] running on GCE node
I20251212 02:55:23.399987 13162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251212 02:55:23.400911 13162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251212 02:55:23.402117 13162 hybrid_clock.cc:648] HybridClock initialized: now 1765508123402017 us; error 90 us; skew 500 ppm
I20251212 02:55:23.402417 13162 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251212 02:55:23.404592 13162 webserver.cc:492] Webserver started at http://127.12.17.195:39585/ using document root <none> and password file <none>
I20251212 02:55:23.405388 13162 fs_manager.cc:362] Metadata directory not provided
I20251212 02:55:23.405498 13162 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 02:55:23.405848 13162 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 02:55:23.408241 13162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/data/instance:
uuid: "5f54f877d2864fd48526ff4e6b01b9ca"
format_stamp: "Formatted at 2025-12-12 02:55:23 on dist-test-slave-8cz7"
I20251212 02:55:23.409065 13162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/wal/instance:
uuid: "5f54f877d2864fd48526ff4e6b01b9ca"
format_stamp: "Formatted at 2025-12-12 02:55:23 on dist-test-slave-8cz7"
I20251212 02:55:23.414423 13162 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251212 02:55:23.418071 13184 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:23.419457 13162 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251212 02:55:23.419605 13162 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/wal
uuid: "5f54f877d2864fd48526ff4e6b01b9ca"
format_stamp: "Formatted at 2025-12-12 02:55:23 on dist-test-slave-8cz7"
I20251212 02:55:23.419786 13162 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 02:55:23.444689 13162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 02:55:23.445554 13162 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 02:55:23.445819 13162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 02:55:23.446743 13162 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 02:55:23.448383 13162 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251212 02:55:23.448469 13162 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:23.448552 13162 ts_tablet_manager.cc:616] Registered 0 tablets
I20251212 02:55:23.448632 13162 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:23.478539 13009 heartbeater.cc:499] Master 127.12.17.253:46623 was elected leader, sending a full tablet report...
I20251212 02:55:23.482182 13162 rpc_server.cc:307] RPC server started. Bound to: 127.12.17.195:42927
I20251212 02:55:23.482246 13296 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.17.195:42927 every 8 connection(s)
I20251212 02:55:23.483678 13162 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/data/info.pb
I20251212 02:55:23.491063 12359 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu as pid 13162
I20251212 02:55:23.491281 12359 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/wal/instance
I20251212 02:55:23.505924 13299 heartbeater.cc:344] Connected to a master server at 127.12.17.253:46623
I20251212 02:55:23.506037 13298 heartbeater.cc:344] Connected to a master server at 127.12.17.254:40649
I20251212 02:55:23.506201 13297 heartbeater.cc:344] Connected to a master server at 127.12.17.252:44417
I20251212 02:55:23.506402 13298 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:23.506404 13299 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:23.506402 13297 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:23.507572 13299 heartbeater.cc:507] Master 127.12.17.253:46623 requested a full tablet report, sending...
I20251212 02:55:23.507607 13297 heartbeater.cc:507] Master 127.12.17.252:44417 requested a full tablet report, sending...
I20251212 02:55:23.507648 13298 heartbeater.cc:507] Master 127.12.17.254:40649 requested a full tablet report, sending...
I20251212 02:55:23.509274 12703 ts_manager.cc:194] Registered new tserver with Master: 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:23.509431 12633 ts_manager.cc:194] Registered new tserver with Master: 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:23.509567 12773 ts_manager.cc:194] Registered new tserver with Master: 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:23.510494 12703 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.12.17.195:47787
I20251212 02:55:23.517477 12359 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251212 02:55:23.548187 13155 heartbeater.cc:499] Master 127.12.17.253:46623 was elected leader, sending a full tablet report...
I20251212 02:55:23.551092 12848 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251212 02:55:23.553972 12698 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53858:
name: "default.Table_2a0b5225731647628e4f9ecd5a0ffb2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.548614 12702 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53858:
name: "default.Table_a63f5d536bfd491fb1d666840afc41a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.548626 12703 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53858:
name: "default.Table_70d527cb5d6a46c0b3a5a994b5db9375"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.553248 12699 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53858:
name: "default.Table_050b838a0a9049a39bee3964eb709425"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.552871 12700 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53858:
name: "default.Table_aeb57448bc884b4ca11ecdd045932bc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.553516 12701 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53858:
name: "default.Table_fc25d2c6f4ad42958441b5b6c7eab15c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.559273 12697 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53858:
name: "default.Table_4f27d7a3eec743a587a57dd44c7c649c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.561599 12695 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53858:
name: "default.Table_f6ccb5471f934cf0a9ff7e662f65b997"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.561496 12696 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53858:
name: "default.Table_b36fce53138b45fc9fb1a2d7ea2afae7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.563269 12694 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53858:
name: "default.Table_b815b4e2b12a41a9a3e0d97ab076f666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251212 02:55:23.566823 12702 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a63f5d536bfd491fb1d666840afc41a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:23.566816 12698 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2a0b5225731647628e4f9ecd5a0ffb2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:23.571534 12699 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_050b838a0a9049a39bee3964eb709425 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:23.571717 12696 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:23.572676 12694 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b815b4e2b12a41a9a3e0d97ab076f666 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:23.572939 12701 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fc25d2c6f4ad42958441b5b6c7eab15c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:23.573984 12697 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4f27d7a3eec743a587a57dd44c7c649c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:23.574878 12700 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_aeb57448bc884b4ca11ecdd045932bc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:23.576387 12703 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_70d527cb5d6a46c0b3a5a994b5db9375 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:23.577375 12695 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f6ccb5471f934cf0a9ff7e662f65b997 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251212 02:55:23.586375 12359 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu pid 12669
I20251212 02:55:23.588187 12733 generic_service.cc:196] Checking for leaks (request via RPC)
I20251212 02:55:23.758195 12359 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu with pid 12669
W20251212 02:55:23.785224 13304 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.17.253:46623)
W20251212 02:55:23.794976 13308 negotiation.cc:337] Failed RPC negotiation. Trace:
1212 02:55:23.786905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.17.253:46623 (local address 127.0.0.1:53860)
1212 02:55:23.789863 (+ 2958us) negotiation.cc:107] Waiting for socket to connect
1212 02:55:23.789889 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1212 02:55:23.790081 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1212 02:55:23.793265 (+ 3184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2668}
I20251212 02:55:23.875782 13039 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ad856a8b02154355b5da573e8fffcb16)
I20251212 02:55:23.875994 13039 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:23.877789 13039 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
I20251212 02:55:23.877799 12783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ed31a99dac451981f4562901f11e87" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7823e6e682ac46a09f968e0742506d6e" is_pre_election: true
I20251212 02:55:23.878039 12783 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62ed31a99dac451981f4562901f11e87 in term 2.
I20251212 02:55:23.878840 12618 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e; no voters:
I20251212 02:55:23.879145 13039 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251212 02:55:23.879271 13039 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ad856a8b02154355b5da573e8fffcb16)
I20251212 02:55:23.879400 13039 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 2 FOLLOWER]: Advancing to term 3
I20251212 02:55:23.883347 13039 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:23.885535 13039 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 3 election: Requested vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
I20251212 02:55:23.886803 12783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ed31a99dac451981f4562901f11e87" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7823e6e682ac46a09f968e0742506d6e"
I20251212 02:55:23.887048 12783 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 2 FOLLOWER]: Advancing to term 3
I20251212 02:55:23.890849 12783 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62ed31a99dac451981f4562901f11e87 in term 3.
I20251212 02:55:23.891399 12618 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e; no voters:
I20251212 02:55:23.892282 13039 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 3 FOLLOWER]: Leader election won for term 3
I20251212 02:55:23.895758 13039 raft_consensus.cc:697] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 3 LEADER]: Becoming Leader. State: Replica: 62ed31a99dac451981f4562901f11e87, State: Running, Role: LEADER
I20251212 02:55:23.896236 13039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
W20251212 02:55:23.897579 12617 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:23.898486 13345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ed31a99dac451981f4562901f11e87. Latest consensus state: current_term: 3 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
W20251212 02:55:23.898830 12617 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:23.898917 13345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: LEADER
I20251212 02:55:23.900410 13347 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251212 02:55:23.902886 13347 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251212 02:55:23.904127 13347 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:23.904229 13347 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251212 02:55:23.906173 13347 catalog_manager.cc:1514] Loading token signing keys...
I20251212 02:55:23.907392 13347 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Loaded TSK: 0
I20251212 02:55:23.908357 13347 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251212 02:55:23.909437 13347 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251212 02:55:23.910523 13347 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251212 02:55:23.923969 12632 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51176:
name: "default.Table_f6ccb5471f934cf0a9ff7e662f65b997"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.924032 12631 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51176:
name: "default.Table_b36fce53138b45fc9fb1a2d7ea2afae7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.924479 12625 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51176:
name: "default.Table_050b838a0a9049a39bee3964eb709425"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.924549 12629 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51176:
name: "default.Table_a63f5d536bfd491fb1d666840afc41a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.925113 12627 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51176:
name: "default.Table_fc25d2c6f4ad42958441b5b6c7eab15c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.925211 12630 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51176:
name: "default.Table_4f27d7a3eec743a587a57dd44c7c649c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.925891 12624 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51176:
name: "default.Table_2a0b5225731647628e4f9ecd5a0ffb2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.926100 12633 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51176:
name: "default.Table_b815b4e2b12a41a9a3e0d97ab076f666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.926616 12626 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51176:
name: "default.Table_aeb57448bc884b4ca11ecdd045932bc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:23.927273 12628 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51176:
name: "default.Table_70d527cb5d6a46c0b3a5a994b5db9375"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251212 02:55:23.932396 12625 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_050b838a0a9049a39bee3964eb709425 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:23.932484 12632 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f6ccb5471f934cf0a9ff7e662f65b997 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:23.934325 12631 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:23.983700 13009 heartbeater.cc:646] Failed to heartbeat to 127.12.17.253:46623 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:24.013088 13299 heartbeater.cc:499] Master 127.12.17.253:46623 was elected leader, sending a full tablet report...
W20251212 02:55:24.014847 13299 heartbeater.cc:646] Failed to heartbeat to 127.12.17.253:46623 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:55:24.067673 13155 heartbeater.cc:646] Failed to heartbeat to 127.12.17.253:46623 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:24.102888 12632 catalog_manager.cc:2174] created HMS catalog entry for table [id=45586ce12b354c4cac23e4a22aafe67c]
I20251212 02:55:24.106694 12783 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 3 FOLLOWER]: Refusing update from remote peer 62ed31a99dac451981f4562901f11e87: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20251212 02:55:24.107398 13345 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20251212 02:55:24.109328 12617 consensus_peers.cc:597] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 -> Peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Couldn't send request to peer ad856a8b02154355b5da573e8fffcb16. Status: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251212 02:55:24.125578 13352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ed31a99dac451981f4562901f11e87. Latest consensus state: current_term: 3 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:24.125859 13352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:24.131196 13345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:24.131466 13345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: LEADER
I20251212 02:55:24.140069 13353 mvcc.cc:204] Tried to move back new op lower bound from 7231521276334321664 to 7231521275483947008. Current Snapshot: MvccSnapshot[applied={T|T < 7231521276334321664}]
I20251212 02:55:24.140918 12632 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51176:
table { table_name: "default.Table_f6ccb5471f934cf0a9ff7e662f65b997" } new_table_name: "default.Table_aad0d00ededb42ff96b1d27e16607697" modify_external_catalogs: true
I20251212 02:55:24.144634 12625 catalog_manager.cc:2174] created HMS catalog entry for table [id=c45bc9e9db5e40a393a6f4caab827f4e]
I20251212 02:55:24.146939 13352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:24.147173 13352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:24.180896 12631 catalog_manager.cc:2174] created HMS catalog entry for table [id=b85eee00b133494989877915a4814803]
I20251212 02:55:24.190590 12625 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51176:
table { table_name: "default.Table_050b838a0a9049a39bee3964eb709425" } new_table_name: "default.Table_7325f02ad24a472ba00bf02a83e75f1b" modify_external_catalogs: true
W20251212 02:55:24.195509 12629 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a63f5d536bfd491fb1d666840afc41a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:24.195636 12630 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4f27d7a3eec743a587a57dd44c7c649c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:24.196758 12627 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fc25d2c6f4ad42958441b5b6c7eab15c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:24.197788 12633 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b815b4e2b12a41a9a3e0d97ab076f666 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:24.198638 12624 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2a0b5225731647628e4f9ecd5a0ffb2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:24.199509 12626 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_aeb57448bc884b4ca11ecdd045932bc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251212 02:55:24.202425 12628 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_70d527cb5d6a46c0b3a5a994b5db9375 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
F20251212 02:55:24.207448 12846 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1765508123, eventType=CREATE_TABLE, dbName=default, tableName=table_f6ccb5471f934cf0a9ff7e662f65b997, message={"server":"","servicePrincipal":"","db":"default","table":"table_f6ccb5471f934cf0a9ff7e662f65b997","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f6ccb5471f934cf0a9ff7e662f65b997\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765508123},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/warehouse/table_f6ccb5471f934cf0a9ff7e662f65b997\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f6ccb5471f934cf0a9ff7e662f65b997\",\"kudu.table_id\":\"45586ce12b354c4cac23e4a22aafe67c\",\"transient_lastDdlTime\":\"1765508123\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cbec40b29e204890941f676300eb9cd8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765508123,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765508123, eventType=CREATE_TABLE, dbName=default, tableName=table_a63f5d536bfd491fb1d666840afc41a4, message={"server":"","servicePrincipal":"","db":"default","table":"table_a63f5d536bfd491fb1d666840afc41a4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a63f5d536bfd491fb1d666840afc41a4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765508123},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/warehouse/table_a63f5d536bfd491fb1d666840afc41a4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a63f5d536bfd491fb1d666840afc41a4\",\"kudu.table_id\":\"6f9b849174f44f8b94870d02244aeba0\",\"transient_lastDdlTime\":\"1765508123\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cbec40b29e204890941f676300eb9cd8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765508123,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fbdec13bdcd google::LogMessage::Fail() at ??:0
@ 0x7fbdec13fb93 google::LogMessage::SendToLog() at ??:0
I20251212 02:55:24.213073 12631 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51176:
table { table_name: "default.Table_b36fce53138b45fc9fb1a2d7ea2afae7" } new_table_name: "default.Table_2f6246f54e0e491287fe6061611a2ca9" modify_external_catalogs: true
@ 0x7fbdec13b7cc google::LogMessage::Flush() at ??:0
@ 0x7fbdec13ba11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fbe02b02aa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251212 02:55:24.229599 13231 tablet_service.cc:1505] Processing CreateTablet for tablet 9bf247fbaba4447c9846db800a856dcd (DEFAULT_TABLE table=default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:24.231423 13231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf247fbaba4447c9846db800a856dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:24.234642 13232 tablet_service.cc:1505] Processing CreateTablet for tablet f5f45a962b144612bda2cd788c22da2c (DEFAULT_TABLE table=default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:24.235210 13232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f45a962b144612bda2cd788c22da2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:24.226365 13230 tablet_service.cc:1505] Processing CreateTablet for tablet 5b7a8d8cef184c40bb6d5c6a26bc137a (DEFAULT_TABLE table=default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:24.236179 13230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b7a8d8cef184c40bb6d5c6a26bc137a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:24.239881 13088 tablet_service.cc:1505] Processing CreateTablet for tablet f5f45a962b144612bda2cd788c22da2c (DEFAULT_TABLE table=default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:24.241601 13088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f45a962b144612bda2cd788c22da2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:24.246022 13087 tablet_service.cc:1505] Processing CreateTablet for tablet 9bf247fbaba4447c9846db800a856dcd (DEFAULT_TABLE table=default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:24.246568 13087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf247fbaba4447c9846db800a856dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:24.249655 13086 tablet_service.cc:1505] Processing CreateTablet for tablet 5b7a8d8cef184c40bb6d5c6a26bc137a (DEFAULT_TABLE table=default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:24.249557 12940 tablet_service.cc:1505] Processing CreateTablet for tablet 5b7a8d8cef184c40bb6d5c6a26bc137a (DEFAULT_TABLE table=default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:24.250203 13086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b7a8d8cef184c40bb6d5c6a26bc137a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:24.250977 12940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b7a8d8cef184c40bb6d5c6a26bc137a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:24.248970 12941 tablet_service.cc:1505] Processing CreateTablet for tablet 9bf247fbaba4447c9846db800a856dcd (DEFAULT_TABLE table=default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:24.253610 12942 tablet_service.cc:1505] Processing CreateTablet for tablet f5f45a962b144612bda2cd788c22da2c (DEFAULT_TABLE table=default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:24.254043 12941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf247fbaba4447c9846db800a856dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:24.254280 12942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f45a962b144612bda2cd788c22da2c. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fbe02b01351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fbe02b08fcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fbe02b08e49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7fbdeeeb03f9 kudu::Thread::SuperviseThread() at ??:0
I20251212 02:55:24.271389 13368 tablet_bootstrap.cc:492] T f5f45a962b144612bda2cd788c22da2c P 480bcf5c1f394464b302900a89ca5bdf: Bootstrap starting.
@ 0x7fbdf24bb6db start_thread at ??:0
@ 0x7fbdeafec71f clone at ??:0
I20251212 02:55:24.278515 13367 tablet_bootstrap.cc:492] T 9bf247fbaba4447c9846db800a856dcd P 8240d2efd35f443a9515f0db38f09f75: Bootstrap starting.
I20251212 02:55:24.282737 13367 tablet_bootstrap.cc:654] T 9bf247fbaba4447c9846db800a856dcd P 8240d2efd35f443a9515f0db38f09f75: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:24.284317 13367 log.cc:826] T 9bf247fbaba4447c9846db800a856dcd P 8240d2efd35f443a9515f0db38f09f75: Log is configured to *not* fsync() on all Append() calls
I20251212 02:55:24.285668 13368 tablet_bootstrap.cc:654] T f5f45a962b144612bda2cd788c22da2c P 480bcf5c1f394464b302900a89ca5bdf: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:24.287038 13368 log.cc:826] T f5f45a962b144612bda2cd788c22da2c P 480bcf5c1f394464b302900a89ca5bdf: Log is configured to *not* fsync() on all Append() calls
I20251212 02:55:24.290019 13367 tablet_bootstrap.cc:492] T 9bf247fbaba4447c9846db800a856dcd P 8240d2efd35f443a9515f0db38f09f75: No bootstrap required, opened a new log
I20251212 02:55:24.290331 13367 ts_tablet_manager.cc:1403] T 9bf247fbaba4447c9846db800a856dcd P 8240d2efd35f443a9515f0db38f09f75: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20251212 02:55:24.301700 13368 tablet_bootstrap.cc:492] T f5f45a962b144612bda2cd788c22da2c P 480bcf5c1f394464b302900a89ca5bdf: No bootstrap required, opened a new log
I20251212 02:55:24.302126 13368 ts_tablet_manager.cc:1403] T f5f45a962b144612bda2cd788c22da2c P 480bcf5c1f394464b302900a89ca5bdf: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.000s
I20251212 02:55:24.306641 13367 raft_consensus.cc:359] T 9bf247fbaba4447c9846db800a856dcd P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.307104 13367 raft_consensus.cc:385] T 9bf247fbaba4447c9846db800a856dcd P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:24.307219 13367 raft_consensus.cc:740] T 9bf247fbaba4447c9846db800a856dcd P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8240d2efd35f443a9515f0db38f09f75, State: Initialized, Role: FOLLOWER
I20251212 02:55:24.307955 13367 consensus_queue.cc:260] T 9bf247fbaba4447c9846db800a856dcd P 8240d2efd35f443a9515f0db38f09f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.313571 13367 ts_tablet_manager.cc:1434] T 9bf247fbaba4447c9846db800a856dcd P 8240d2efd35f443a9515f0db38f09f75: Time spent starting tablet: real 0.023s user 0.012s sys 0.000s
I20251212 02:55:24.314040 13367 tablet_bootstrap.cc:492] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75: Bootstrap starting.
I20251212 02:55:24.314608 13154 heartbeater.cc:499] Master 127.12.17.254:40649 was elected leader, sending a full tablet report...
I20251212 02:55:24.316020 13367 tablet_bootstrap.cc:654] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:24.316640 13368 raft_consensus.cc:359] T f5f45a962b144612bda2cd788c22da2c P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:24.317090 13368 raft_consensus.cc:385] T f5f45a962b144612bda2cd788c22da2c P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:24.317222 13368 raft_consensus.cc:740] T f5f45a962b144612bda2cd788c22da2c P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 480bcf5c1f394464b302900a89ca5bdf, State: Initialized, Role: FOLLOWER
I20251212 02:55:24.318101 13368 consensus_queue.cc:260] T f5f45a962b144612bda2cd788c22da2c P 480bcf5c1f394464b302900a89ca5bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:24.319299 13374 tablet_bootstrap.cc:492] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca: Bootstrap starting.
I20251212 02:55:24.323518 13008 heartbeater.cc:499] Master 127.12.17.254:40649 was elected leader, sending a full tablet report...
I20251212 02:55:24.335888 13368 ts_tablet_manager.cc:1434] T f5f45a962b144612bda2cd788c22da2c P 480bcf5c1f394464b302900a89ca5bdf: Time spent starting tablet: real 0.034s user 0.008s sys 0.004s
I20251212 02:55:24.336635 13374 tablet_bootstrap.cc:654] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:24.338018 13367 tablet_bootstrap.cc:492] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75: No bootstrap required, opened a new log
I20251212 02:55:24.338102 13374 log.cc:826] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca: Log is configured to *not* fsync() on all Append() calls
I20251212 02:55:24.338196 13367 ts_tablet_manager.cc:1403] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.008s
I20251212 02:55:24.339195 13367 raft_consensus.cc:359] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.339383 13367 raft_consensus.cc:385] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:24.339470 13367 raft_consensus.cc:740] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8240d2efd35f443a9515f0db38f09f75, State: Initialized, Role: FOLLOWER
I20251212 02:55:24.339759 13367 consensus_queue.cc:260] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.341274 13368 tablet_bootstrap.cc:492] T 9bf247fbaba4447c9846db800a856dcd P 480bcf5c1f394464b302900a89ca5bdf: Bootstrap starting.
I20251212 02:55:24.342345 13367 ts_tablet_manager.cc:1434] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20251212 02:55:24.343531 13367 tablet_bootstrap.cc:492] T f5f45a962b144612bda2cd788c22da2c P 8240d2efd35f443a9515f0db38f09f75: Bootstrap starting.
I20251212 02:55:24.345264 13367 tablet_bootstrap.cc:654] T f5f45a962b144612bda2cd788c22da2c P 8240d2efd35f443a9515f0db38f09f75: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:24.347060 13368 tablet_bootstrap.cc:654] T 9bf247fbaba4447c9846db800a856dcd P 480bcf5c1f394464b302900a89ca5bdf: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:24.350865 13374 tablet_bootstrap.cc:492] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca: No bootstrap required, opened a new log
I20251212 02:55:24.351174 13374 ts_tablet_manager.cc:1403] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.006s
I20251212 02:55:24.355867 13367 tablet_bootstrap.cc:492] T f5f45a962b144612bda2cd788c22da2c P 8240d2efd35f443a9515f0db38f09f75: No bootstrap required, opened a new log
I20251212 02:55:24.356045 13367 ts_tablet_manager.cc:1403] T f5f45a962b144612bda2cd788c22da2c P 8240d2efd35f443a9515f0db38f09f75: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20251212 02:55:24.356828 13367 raft_consensus.cc:359] T f5f45a962b144612bda2cd788c22da2c P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:24.357021 13367 raft_consensus.cc:385] T f5f45a962b144612bda2cd788c22da2c P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:24.357096 13367 raft_consensus.cc:740] T f5f45a962b144612bda2cd788c22da2c P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8240d2efd35f443a9515f0db38f09f75, State: Initialized, Role: FOLLOWER
I20251212 02:55:24.357389 13367 consensus_queue.cc:260] T f5f45a962b144612bda2cd788c22da2c P 8240d2efd35f443a9515f0db38f09f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:24.359042 13367 ts_tablet_manager.cc:1434] T f5f45a962b144612bda2cd788c22da2c P 8240d2efd35f443a9515f0db38f09f75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251212 02:55:24.361945 13368 tablet_bootstrap.cc:492] T 9bf247fbaba4447c9846db800a856dcd P 480bcf5c1f394464b302900a89ca5bdf: No bootstrap required, opened a new log
I20251212 02:55:24.362099 13368 ts_tablet_manager.cc:1403] T 9bf247fbaba4447c9846db800a856dcd P 480bcf5c1f394464b302900a89ca5bdf: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20251212 02:55:24.363063 13368 raft_consensus.cc:359] T 9bf247fbaba4447c9846db800a856dcd P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.363247 13368 raft_consensus.cc:385] T 9bf247fbaba4447c9846db800a856dcd P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:24.363330 13368 raft_consensus.cc:740] T 9bf247fbaba4447c9846db800a856dcd P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 480bcf5c1f394464b302900a89ca5bdf, State: Initialized, Role: FOLLOWER
I20251212 02:55:24.363610 13368 consensus_queue.cc:260] T 9bf247fbaba4447c9846db800a856dcd P 480bcf5c1f394464b302900a89ca5bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.366613 13374 raft_consensus.cc:359] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:24.366988 13368 ts_tablet_manager.cc:1434] T 9bf247fbaba4447c9846db800a856dcd P 480bcf5c1f394464b302900a89ca5bdf: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251212 02:55:24.367049 13374 raft_consensus.cc:385] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:24.367152 13374 raft_consensus.cc:740] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f54f877d2864fd48526ff4e6b01b9ca, State: Initialized, Role: FOLLOWER
I20251212 02:55:24.370746 13368 tablet_bootstrap.cc:492] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf: Bootstrap starting.
I20251212 02:55:24.372457 13368 tablet_bootstrap.cc:654] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:24.377948 13374 consensus_queue.cc:260] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:24.382383 13374 ts_tablet_manager.cc:1434] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca: Time spent starting tablet: real 0.031s user 0.010s sys 0.004s
I20251212 02:55:24.389125 13298 heartbeater.cc:499] Master 127.12.17.254:40649 was elected leader, sending a full tablet report...
I20251212 02:55:24.391309 13374 tablet_bootstrap.cc:492] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca: Bootstrap starting.
I20251212 02:55:24.393778 13374 tablet_bootstrap.cc:654] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:24.397554 13368 tablet_bootstrap.cc:492] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf: No bootstrap required, opened a new log
I20251212 02:55:24.397745 13368 ts_tablet_manager.cc:1403] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.013s
I20251212 02:55:24.398556 13368 raft_consensus.cc:359] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.398761 13368 raft_consensus.cc:385] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:24.398847 13368 raft_consensus.cc:740] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 480bcf5c1f394464b302900a89ca5bdf, State: Initialized, Role: FOLLOWER
I20251212 02:55:24.399130 13368 consensus_queue.cc:260] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.402521 13368 ts_tablet_manager.cc:1434] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251212 02:55:24.405436 13374 tablet_bootstrap.cc:492] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca: No bootstrap required, opened a new log
I20251212 02:55:24.405568 13374 ts_tablet_manager.cc:1403] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20251212 02:55:24.406363 13374 raft_consensus.cc:359] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.406502 13374 raft_consensus.cc:385] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:24.406559 13374 raft_consensus.cc:740] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f54f877d2864fd48526ff4e6b01b9ca, State: Initialized, Role: FOLLOWER
I20251212 02:55:24.406792 13374 consensus_queue.cc:260] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.407266 13374 ts_tablet_manager.cc:1434] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251212 02:55:24.408672 13374 tablet_bootstrap.cc:492] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca: Bootstrap starting.
I20251212 02:55:24.410529 13374 tablet_bootstrap.cc:654] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:24.423528 13374 tablet_bootstrap.cc:492] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca: No bootstrap required, opened a new log
I20251212 02:55:24.423705 13374 ts_tablet_manager.cc:1403] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.002s
I20251212 02:55:24.424448 13374 raft_consensus.cc:359] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.424651 13374 raft_consensus.cc:385] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:24.424733 13374 raft_consensus.cc:740] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f54f877d2864fd48526ff4e6b01b9ca, State: Initialized, Role: FOLLOWER
I20251212 02:55:24.424993 13374 consensus_queue.cc:260] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.425519 13374 ts_tablet_manager.cc:1434] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251212 02:55:24.442682 13010 tablet.cc:2378] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:55:24.474 [HMS - ERROR - pool-6-thread-2] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_a63f5d536bfd491fb1d666840afc41a4 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20251212 02:55:24.493031 13300 tablet.cc:2378] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251212 02:55:24.505812 13156 tablet.cc:2378] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251212 02:55:24.551724 12359 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
/tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.253:46623
--webserver_interface=127.12.17.253
--webserver_port=36499
--builtin_ntp_servers=127.12.17.212:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251212 02:55:24.595156 12898 connection.cc:537] server connection from 127.0.0.1:37282 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:24.595156 13041 connection.cc:537] client connection to 127.12.17.254:40649 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:24.595228 13304 connection.cc:537] client connection to 127.12.17.254:40649 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:24.595341 13043 connection.cc:537] server connection from 127.0.0.1:37828 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:24.595626 13188 connection.cc:537] server connection from 127.0.0.1:42934 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:24.595959 12895 connection.cc:537] client connection to 127.12.17.254:40649 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:24.596889 13304 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.17.254:40649)
W20251212 02:55:24.596940 13185 connection.cc:537] client connection to 127.12.17.254:40649 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:24.597993 13350 negotiation.cc:337] Failed RPC negotiation. Trace:
1212 02:55:24.596206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.17.254:40649 (local address 127.12.17.194:48601)
1212 02:55:24.596480 (+ 274us) negotiation.cc:107] Waiting for socket to connect
1212 02:55:24.596500 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1212 02:55:24.596653 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1212 02:55:24.597064 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":157}
W20251212 02:55:24.598062 13304 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.17.254:40649)
I20251212 02:55:24.677407 13384 raft_consensus.cc:493] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:24.677678 13384 raft_consensus.cc:515] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:24.679510 13384 leader_election.cc:290] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431), 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
I20251212 02:55:24.687943 12962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5f45a962b144612bda2cd788c22da2c" candidate_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "480bcf5c1f394464b302900a89ca5bdf" is_pre_election: true
I20251212 02:55:24.688499 12962 raft_consensus.cc:2468] T f5f45a962b144612bda2cd788c22da2c P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f54f877d2864fd48526ff4e6b01b9ca in term 0.
I20251212 02:55:24.688446 13108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5f45a962b144612bda2cd788c22da2c" candidate_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8240d2efd35f443a9515f0db38f09f75" is_pre_election: true
I20251212 02:55:24.688863 13108 raft_consensus.cc:2468] T f5f45a962b144612bda2cd788c22da2c P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f54f877d2864fd48526ff4e6b01b9ca in term 0.
I20251212 02:55:24.689250 13187 leader_election.cc:304] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 480bcf5c1f394464b302900a89ca5bdf, 5f54f877d2864fd48526ff4e6b01b9ca; no voters:
I20251212 02:55:24.689821 13384 raft_consensus.cc:2804] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251212 02:55:24.689985 13384 raft_consensus.cc:493] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 02:55:24.690097 13384 raft_consensus.cc:3060] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:24.692162 13384 raft_consensus.cc:515] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:24.692778 13384 leader_election.cc:290] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [CANDIDATE]: Term 1 election: Requested vote from peers 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431), 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
I20251212 02:55:24.693462 12962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5f45a962b144612bda2cd788c22da2c" candidate_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "480bcf5c1f394464b302900a89ca5bdf"
I20251212 02:55:24.693545 13108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5f45a962b144612bda2cd788c22da2c" candidate_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8240d2efd35f443a9515f0db38f09f75"
I20251212 02:55:24.693674 12962 raft_consensus.cc:3060] T f5f45a962b144612bda2cd788c22da2c P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:24.693758 13108 raft_consensus.cc:3060] T f5f45a962b144612bda2cd788c22da2c P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:24.696278 13108 raft_consensus.cc:2468] T f5f45a962b144612bda2cd788c22da2c P 8240d2efd35f443a9515f0db38f09f75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f54f877d2864fd48526ff4e6b01b9ca in term 1.
I20251212 02:55:24.696853 13186 leader_election.cc:304] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f54f877d2864fd48526ff4e6b01b9ca, 8240d2efd35f443a9515f0db38f09f75; no voters:
I20251212 02:55:24.697328 13384 raft_consensus.cc:2804] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [term 1 FOLLOWER]: Leader election won for term 1
I20251212 02:55:24.697948 13384 raft_consensus.cc:697] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [term 1 LEADER]: Becoming Leader. State: Replica: 5f54f877d2864fd48526ff4e6b01b9ca, State: Running, Role: LEADER
I20251212 02:55:24.698509 13384 consensus_queue.cc:237] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:24.699098 12962 raft_consensus.cc:2468] T f5f45a962b144612bda2cd788c22da2c P 480bcf5c1f394464b302900a89ca5bdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f54f877d2864fd48526ff4e6b01b9ca in term 1.
I20251212 02:55:24.718182 13396 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62ed31a99dac451981f4562901f11e87)
I20251212 02:55:24.718350 13396 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:24.719003 13396 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
W20251212 02:55:24.720201 12757 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:24.722076 12757 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:24.722250 12757 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:24.722391 12757 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e; no voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:24.722657 13396 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251212 02:55:24.784075 13375 raft_consensus.cc:493] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:24.784307 13375 raft_consensus.cc:515] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.786087 13375 leader_election.cc:290] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431), 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:24.793084 12962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b7a8d8cef184c40bb6d5c6a26bc137a" candidate_uuid: "8240d2efd35f443a9515f0db38f09f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "480bcf5c1f394464b302900a89ca5bdf" is_pre_election: true
I20251212 02:55:24.793401 12962 raft_consensus.cc:2468] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8240d2efd35f443a9515f0db38f09f75 in term 0.
I20251212 02:55:24.793921 13043 leader_election.cc:304] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 480bcf5c1f394464b302900a89ca5bdf, 8240d2efd35f443a9515f0db38f09f75; no voters:
I20251212 02:55:24.794414 13375 raft_consensus.cc:2804] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251212 02:55:24.794641 13375 raft_consensus.cc:493] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 02:55:24.794751 13375 raft_consensus.cc:3060] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:24.794541 13252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b7a8d8cef184c40bb6d5c6a26bc137a" candidate_uuid: "8240d2efd35f443a9515f0db38f09f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" is_pre_election: true
I20251212 02:55:24.795092 13252 raft_consensus.cc:2468] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8240d2efd35f443a9515f0db38f09f75 in term 0.
I20251212 02:55:24.797089 13375 raft_consensus.cc:515] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.797667 13375 leader_election.cc:290] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [CANDIDATE]: Term 1 election: Requested vote from peers 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431), 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:24.798158 12962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b7a8d8cef184c40bb6d5c6a26bc137a" candidate_uuid: "8240d2efd35f443a9515f0db38f09f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "480bcf5c1f394464b302900a89ca5bdf"
I20251212 02:55:24.798311 13252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b7a8d8cef184c40bb6d5c6a26bc137a" candidate_uuid: "8240d2efd35f443a9515f0db38f09f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f54f877d2864fd48526ff4e6b01b9ca"
I20251212 02:55:24.798386 12962 raft_consensus.cc:3060] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:24.798514 13252 raft_consensus.cc:3060] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:24.800675 13252 raft_consensus.cc:2468] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8240d2efd35f443a9515f0db38f09f75 in term 1.
I20251212 02:55:24.801120 13043 leader_election.cc:304] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f54f877d2864fd48526ff4e6b01b9ca, 8240d2efd35f443a9515f0db38f09f75; no voters:
I20251212 02:55:24.801275 12962 raft_consensus.cc:2468] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8240d2efd35f443a9515f0db38f09f75 in term 1.
I20251212 02:55:24.801443 13375 raft_consensus.cc:2804] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [term 1 FOLLOWER]: Leader election won for term 1
I20251212 02:55:24.801906 13375 raft_consensus.cc:697] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [term 1 LEADER]: Becoming Leader. State: Replica: 8240d2efd35f443a9515f0db38f09f75, State: Running, Role: LEADER
I20251212 02:55:24.802760 13375 consensus_queue.cc:237] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.902500 13384 raft_consensus.cc:493] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:24.902734 13384 raft_consensus.cc:515] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.903520 13384 leader_election.cc:290] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431), 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
I20251212 02:55:24.904111 13108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bf247fbaba4447c9846db800a856dcd" candidate_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8240d2efd35f443a9515f0db38f09f75" is_pre_election: true
I20251212 02:55:24.904142 12962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bf247fbaba4447c9846db800a856dcd" candidate_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "480bcf5c1f394464b302900a89ca5bdf" is_pre_election: true
I20251212 02:55:24.904352 13108 raft_consensus.cc:2468] T 9bf247fbaba4447c9846db800a856dcd P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f54f877d2864fd48526ff4e6b01b9ca in term 0.
I20251212 02:55:24.904407 12962 raft_consensus.cc:2468] T 9bf247fbaba4447c9846db800a856dcd P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f54f877d2864fd48526ff4e6b01b9ca in term 0.
I20251212 02:55:24.904881 13187 leader_election.cc:304] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 480bcf5c1f394464b302900a89ca5bdf, 5f54f877d2864fd48526ff4e6b01b9ca, 8240d2efd35f443a9515f0db38f09f75; no voters:
I20251212 02:55:24.905192 13384 raft_consensus.cc:2804] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251212 02:55:24.905303 13384 raft_consensus.cc:493] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 02:55:24.905400 13384 raft_consensus.cc:3060] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:24.907523 13384 raft_consensus.cc:515] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:24.908114 13384 leader_election.cc:290] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [CANDIDATE]: Term 1 election: Requested vote from peers 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431), 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
I20251212 02:55:24.908476 12962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bf247fbaba4447c9846db800a856dcd" candidate_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "480bcf5c1f394464b302900a89ca5bdf"
I20251212 02:55:24.908674 12962 raft_consensus.cc:3060] T 9bf247fbaba4447c9846db800a856dcd P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:24.908648 13108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bf247fbaba4447c9846db800a856dcd" candidate_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8240d2efd35f443a9515f0db38f09f75"
I20251212 02:55:24.908861 13108 raft_consensus.cc:3060] T 9bf247fbaba4447c9846db800a856dcd P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:24.911419 12962 raft_consensus.cc:2468] T 9bf247fbaba4447c9846db800a856dcd P 480bcf5c1f394464b302900a89ca5bdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f54f877d2864fd48526ff4e6b01b9ca in term 1.
I20251212 02:55:24.911520 13108 raft_consensus.cc:2468] T 9bf247fbaba4447c9846db800a856dcd P 8240d2efd35f443a9515f0db38f09f75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f54f877d2864fd48526ff4e6b01b9ca in term 1.
I20251212 02:55:24.912117 13186 leader_election.cc:304] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f54f877d2864fd48526ff4e6b01b9ca, 8240d2efd35f443a9515f0db38f09f75; no voters:
I20251212 02:55:24.912535 13384 raft_consensus.cc:2804] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [term 1 FOLLOWER]: Leader election won for term 1
I20251212 02:55:24.912716 13384 raft_consensus.cc:697] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [term 1 LEADER]: Becoming Leader. State: Replica: 5f54f877d2864fd48526ff4e6b01b9ca, State: Running, Role: LEADER
I20251212 02:55:24.912971 13384 consensus_queue.cc:237] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
W20251212 02:55:24.924779 13390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251212 02:55:24.925145 13390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251212 02:55:24.925295 13390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251212 02:55:24.933833 13390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251212 02:55:24.933951 13390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251212 02:55:24.934052 13390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251212 02:55:24.934128 13390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251212 02:55:24.944717 13390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.17.212:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.253:46623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.17.253
--webserver_port=36499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 12 Dec 2025 02:43:20 UTC on 24a791456cd2
build id 9472
ASAN enabled
I20251212 02:55:24.946496 13390 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 02:55:24.948324 13390 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 02:55:24.956430 13409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:24.956759 13410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:24.957530 13412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:24.958184 13390 server_base.cc:1047] running on GCE node
I20251212 02:55:24.958837 13390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251212 02:55:24.960202 13390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251212 02:55:24.961405 13390 hybrid_clock.cc:648] HybridClock initialized: now 1765508124961360 us; error 36 us; skew 500 ppm
I20251212 02:55:24.961787 13390 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251212 02:55:24.963928 13390 webserver.cc:492] Webserver started at http://127.12.17.253:36499/ using document root <none> and password file <none>
I20251212 02:55:24.964637 13390 fs_manager.cc:362] Metadata directory not provided
I20251212 02:55:24.964737 13390 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 02:55:24.970654 13390 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20251212 02:55:24.973541 13418 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:24.974943 13390 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251212 02:55:24.975121 13390 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data,/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal
uuid: "ad856a8b02154355b5da573e8fffcb16"
format_stamp: "Formatted at 2025-12-12 02:55:20 on dist-test-slave-8cz7"
I20251212 02:55:24.975832 13390 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 02:55:25.004385 13390 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 02:55:25.005306 13390 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 02:55:25.005576 13390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 02:55:25.025758 13390 rpc_server.cc:307] RPC server started. Bound to: 127.12.17.253:46623
I20251212 02:55:25.025802 13469 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.17.253:46623 every 8 connection(s)
I20251212 02:55:25.027911 13390 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data/info.pb
I20251212 02:55:25.034636 12359 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu as pid 13390
I20251212 02:55:25.035495 13470 sys_catalog.cc:263] Verifying existing consensus state
I20251212 02:55:25.040848 13470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Bootstrap starting.
I20251212 02:55:25.049253 13470 log.cc:826] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Log is configured to *not* fsync() on all Append() calls
I20251212 02:55:25.057458 13470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251212 02:55:25.058014 13470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Bootstrap complete.
I20251212 02:55:25.063287 13470 raft_consensus.cc:359] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:25.064019 13470 raft_consensus.cc:740] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad856a8b02154355b5da573e8fffcb16, State: Initialized, Role: FOLLOWER
I20251212 02:55:25.064682 13470 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:25.065549 13475 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:25.065840 13475 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:25.066623 13470 sys_catalog.cc:565] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: configured and running, proceeding with master startup.
I20251212 02:55:25.080463 13489 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:25.080576 13489 catalog_manager.cc:1562] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: loading cluster ID for follower catalog manager: success
I20251212 02:55:25.077665 13390 master_runner.cc:186] Error getting master registration for 127.12.17.254:40649: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111),
I20251212 02:55:25.083891 13390 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:55:25.084976 13489 catalog_manager.cc:1584] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: acquiring CA information for follower catalog manager: success
I20251212 02:55:25.087796 13489 catalog_manager.cc:1612] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251212 02:55:25.312237 13155 heartbeater.cc:344] Connected to a master server at 127.12.17.253:46623
I20251212 02:55:25.316138 13434 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" instance_seqno: 1765508122990947) as {username='slave'} at 127.12.17.194:56241; Asking this server to re-register.
I20251212 02:55:25.316982 13155 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:25.318874 13434 ts_manager.cc:194] Registered new tserver with Master: 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
I20251212 02:55:25.389918 13494 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62ed31a99dac451981f4562901f11e87)
I20251212 02:55:25.390224 13494 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:25.391103 13494 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
W20251212 02:55:25.393493 12757 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
I20251212 02:55:25.399698 13445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7823e6e682ac46a09f968e0742506d6e" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "ad856a8b02154355b5da573e8fffcb16" is_pre_election: true
I20251212 02:55:25.400326 13445 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7823e6e682ac46a09f968e0742506d6e in term 2.
I20251212 02:55:25.400887 12757 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87
I20251212 02:55:25.401249 13494 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251212 02:55:25.401372 13494 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 62ed31a99dac451981f4562901f11e87)
I20251212 02:55:25.401466 13494 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 3 FOLLOWER]: Advancing to term 4
I20251212 02:55:25.403475 13494 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:25.404124 13494 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 4 election: Requested vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
I20251212 02:55:25.404824 13445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7823e6e682ac46a09f968e0742506d6e" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "ad856a8b02154355b5da573e8fffcb16"
I20251212 02:55:25.405042 13445 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 2 FOLLOWER]: Advancing to term 4
W20251212 02:55:25.405416 12757 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
I20251212 02:55:25.412243 13445 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7823e6e682ac46a09f968e0742506d6e in term 4.
I20251212 02:55:25.412758 12757 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87
I20251212 02:55:25.413250 13494 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 4 FOLLOWER]: Leader election won for term 4
I20251212 02:55:25.413789 13494 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 4 LEADER]: Becoming Leader. State: Replica: 7823e6e682ac46a09f968e0742506d6e, State: Running, Role: LEADER
I20251212 02:55:25.414422 13494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:25.415213 13009 heartbeater.cc:344] Connected to a master server at 127.12.17.253:46623
I20251212 02:55:25.416867 13434 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" instance_seqno: 1765508122427264) as {username='slave'} at 127.12.17.193:54803; Asking this server to re-register.
I20251212 02:55:25.417632 13009 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:25.418910 13434 ts_manager.cc:194] Registered new tserver with Master: 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431)
I20251212 02:55:25.427666 13498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7823e6e682ac46a09f968e0742506d6e. Latest consensus state: current_term: 4 leader_uuid: "7823e6e682ac46a09f968e0742506d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:25.428234 13498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: LEADER
I20251212 02:55:25.438257 13501 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251212 02:55:25.439930 13299 heartbeater.cc:344] Connected to a master server at 127.12.17.253:46623
I20251212 02:55:25.440136 13299 heartbeater.cc:499] Master 127.12.17.253:46623 was elected leader, sending a full tablet report...
I20251212 02:55:25.441304 13501 catalog_manager.cc:679] Loaded metadata for table default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c]
I20251212 02:55:25.441540 13434 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" instance_seqno: 1765508123473667) as {username='slave'} at 127.12.17.195:38315; Asking this server to re-register.
I20251212 02:55:25.441875 13501 catalog_manager.cc:679] Loaded metadata for table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]
I20251212 02:55:25.442270 13501 catalog_manager.cc:679] Loaded metadata for table default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e]
I20251212 02:55:25.442407 13299 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:25.443900 13434 ts_manager.cc:194] Registered new tserver with Master: 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:25.444202 13501 tablet_loader.cc:96] loaded metadata for tablet 5b7a8d8cef184c40bb6d5c6a26bc137a (table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803])
I20251212 02:55:25.444628 13501 tablet_loader.cc:96] loaded metadata for tablet 9bf247fbaba4447c9846db800a856dcd (table default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e])
I20251212 02:55:25.445397 13501 tablet_loader.cc:96] loaded metadata for tablet f5f45a962b144612bda2cd788c22da2c (table default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c])
I20251212 02:55:25.445667 13501 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251212 02:55:25.446722 13501 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:25.446815 13501 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251212 02:55:25.448267 13501 catalog_manager.cc:1514] Loading token signing keys...
I20251212 02:55:25.449046 13501 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Loaded TSK: 0
I20251212 02:55:25.449919 13501 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251212 02:55:25.450775 13501 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251212 02:55:25.451493 13501 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251212 02:55:25.479301 12772 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36506:
name: "default.Table_2bed983ea7fa4fd39e17cffd5a24225f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:25.479637 12771 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36506:
name: "default.Table_70d527cb5d6a46c0b3a5a994b5db9375"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:25.479775 12769 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36506:
name: "default.Table_4f27d7a3eec743a587a57dd44c7c649c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:25.480587 12767 master_service.cc:954] ReplaceTablet: received request to replace tablet 5b7a8d8cef184c40bb6d5c6a26bc137a from {username='slave'} at 127.0.0.1:36506
I20251212 02:55:25.479301 12773 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36506:
name: "default.Table_397fb5c3eafe4e49b3667efcf7811f7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:25.480346 12765 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36506:
name: "default.Table_aeb57448bc884b4ca11ecdd045932bc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:25.481496 12770 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36506:
name: "default.Table_a63f5d536bfd491fb1d666840afc41a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:25.482169 12768 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36506:
name: "default.Table_b815b4e2b12a41a9a3e0d97ab076f666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:25.482801 12766 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36506:
name: "default.Table_fc25d2c6f4ad42958441b5b6c7eab15c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:25.483398 12764 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36506:
name: "default.Table_2a0b5225731647628e4f9ecd5a0ffb2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251212 02:55:25.488283 12829 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1765508123, eventType=CREATE_TABLE, dbName=default, tableName=table_f6ccb5471f934cf0a9ff7e662f65b997, message={"server":"","servicePrincipal":"","db":"default","table":"table_f6ccb5471f934cf0a9ff7e662f65b997","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f6ccb5471f934cf0a9ff7e662f65b997\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765508123},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/warehouse/table_f6ccb5471f934cf0a9ff7e662f65b997\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f6ccb5471f934cf0a9ff7e662f65b997\",\"kudu.table_id\":\"45586ce12b354c4cac23e4a22aafe67c\",\"transient_lastDdlTime\":\"1765508123\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cbec40b29e204890941f676300eb9cd8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765508123,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765508123, eventType=CREATE_TABLE, dbName=default, tableName=table_a63f5d536bfd491fb1d666840afc41a4, message={"server":"","servicePrincipal":"","db":"default","table":"table_a63f5d536bfd491fb1d666840afc41a4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a63f5d536bfd491fb1d666840afc41a4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765508123},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/warehouse/table_a63f5d536bfd491fb1d666840afc41a4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a63f5d536bfd491fb1d666840afc41a4\",\"kudu.table_id\":\"6f9b849174f44f8b94870d02244aeba0\",\"transient_lastDdlTime\":\"1765508123\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cbec40b29e204890941f676300eb9cd8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765508123,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fe43d550dcd google::LogMessage::Fail() at ??:0
W20251212 02:55:25.491294 12757 consensus_peers.cc:597] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e -> Peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Couldn't send request to peer 62ed31a99dac451981f4562901f11e87. Status: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251212 02:55:25.492228 13445 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Refusing update from remote peer 7823e6e682ac46a09f968e0742506d6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
@ 0x7fe43d554b93 google::LogMessage::SendToLog() at ??:0
I20251212 02:55:25.493053 13498 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 5, Time since last communication: 0.000s
@ 0x7fe43d5507cc google::LogMessage::Flush() at ??:0
@ 0x7fe43d550a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fe453f17aa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251212 02:55:25.504705 13475 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7823e6e682ac46a09f968e0742506d6e. Latest consensus state: current_term: 4 leader_uuid: "7823e6e682ac46a09f968e0742506d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:25.505017 13475 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:25.505442 13508 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "7823e6e682ac46a09f968e0742506d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:25.505720 13508 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fe453f16351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251212 02:55:25.509651 13508 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "7823e6e682ac46a09f968e0742506d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:25.509913 13508 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:25.515374 12767 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Sending DeleteTablet for 3 replicas of tablet 5b7a8d8cef184c40bb6d5c6a26bc137a
I20251212 02:55:25.517242 13505 mvcc.cc:204] Tried to move back new op lower bound from 7231521281989144576 to 7231521275483947008. Current Snapshot: MvccSnapshot[applied={T|T < 7231521281989144576}]
I20251212 02:55:25.517880 12767 catalog_manager.cc:6724] ReplaceTablet: tablet 5b7a8d8cef184c40bb6d5c6a26bc137a deleted and replaced by tablet ef7a386d14cb4d43b9affd638418d0ca
@ 0x7fe453f1dfcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251212 02:55:25.533598 12767 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36506:
table { table_name: "default.Table_f6ccb5471f934cf0a9ff7e662f65b997" } new_table_name: "default.Table_aad0d00ededb42ff96b1d27e16607697" modify_external_catalogs: true
@ 0x7fe453f1de49 std::_Function_handler<>::_M_invoke() at ??:0
I20251212 02:55:25.535491 13494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "7823e6e682ac46a09f968e0742506d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:25.535894 13494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: LEADER
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7fe4402c53f9 kudu::Thread::SuperviseThread() at ??:0
I20251212 02:55:25.551756 12940 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b7a8d8cef184c40bb6d5c6a26bc137a with delete_type TABLET_DATA_DELETED (replaced by tablet ef7a386d14cb4d43b9affd638418d0ca) from {username='slave'} at 127.0.0.1:37290
I20251212 02:55:25.551887 12941 tablet_service.cc:1505] Processing CreateTablet for tablet ef7a386d14cb4d43b9affd638418d0ca (DEFAULT_TABLE table=default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:25.552388 12941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef7a386d14cb4d43b9affd638418d0ca. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fe4438d06db start_thread at ??:0
I20251212 02:55:25.555047 13232 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b7a8d8cef184c40bb6d5c6a26bc137a with delete_type TABLET_DATA_DELETED (replaced by tablet ef7a386d14cb4d43b9affd638418d0ca) from {username='slave'} at 127.0.0.1:42936
I20251212 02:55:25.555737 13087 tablet_service.cc:1505] Processing CreateTablet for tablet ef7a386d14cb4d43b9affd638418d0ca (DEFAULT_TABLE table=default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:25.555776 13231 tablet_service.cc:1505] Processing CreateTablet for tablet ef7a386d14cb4d43b9affd638418d0ca (DEFAULT_TABLE table=default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:25.555913 13088 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b7a8d8cef184c40bb6d5c6a26bc137a with delete_type TABLET_DATA_DELETED (replaced by tablet ef7a386d14cb4d43b9affd638418d0ca) from {username='slave'} at 127.0.0.1:37836
I20251212 02:55:25.556155 13231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef7a386d14cb4d43b9affd638418d0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:25.556166 13087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef7a386d14cb4d43b9affd638418d0ca. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fe43c40171f clone at ??:0
I20251212 02:55:25.558166 13523 tablet_replica.cc:333] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf: stopping tablet replica
I20251212 02:55:25.558754 13523 raft_consensus.cc:2243] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20251212 02:55:25.559260 13523 raft_consensus.cc:2272] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20251212 02:55:25.561118 13523 ts_tablet_manager.cc:1916] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251212 02:55:25.570842 13523 ts_tablet_manager.cc:1929] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251212 02:55:25.571686 13523 log.cc:1199] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf: Deleting WAL directory at /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/wal/wals/5b7a8d8cef184c40bb6d5c6a26bc137a
I20251212 02:55:25.572304 13523 ts_tablet_manager.cc:1950] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 480bcf5c1f394464b302900a89ca5bdf: Deleting consensus metadata
I20251212 02:55:25.573050 13526 tablet_bootstrap.cc:492] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca: Bootstrap starting.
I20251212 02:55:25.577612 13524 tablet_replica.cc:333] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca: stopping tablet replica
I20251212 02:55:25.578153 13524 raft_consensus.cc:2243] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20251212 02:55:25.578483 13524 raft_consensus.cc:2272] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20251212 02:55:25.580040 13524 ts_tablet_manager.cc:1916] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251212 02:55:25.581590 13528 tablet_bootstrap.cc:492] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf: Bootstrap starting.
I20251212 02:55:25.583515 13525 tablet_replica.cc:333] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75: stopping tablet replica
I20251212 02:55:25.583652 13528 tablet_bootstrap.cc:654] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:25.584021 13526 tablet_bootstrap.cc:654] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:25.584028 13525 raft_consensus.cc:2243] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [term 1 LEADER]: Raft consensus shutting down.
I20251212 02:55:25.584649 13525 pending_rounds.cc:70] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75: Trying to abort 1 pending ops.
I20251212 02:55:25.584805 13525 pending_rounds.cc:77] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7231521279197814784 op_type: NO_OP noop_request { }
I20251212 02:55:25.585076 13525 raft_consensus.cc:2889] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251212 02:55:25.585281 13525 raft_consensus.cc:2272] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251212 02:55:25.585477 13527 tablet_bootstrap.cc:492] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75: Bootstrap starting.
I20251212 02:55:25.587385 13525 ts_tablet_manager.cc:1916] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251212 02:55:25.587512 13527 tablet_bootstrap.cc:654] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:25.598026 13526 tablet_bootstrap.cc:492] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca: No bootstrap required, opened a new log
I20251212 02:55:25.598212 13526 ts_tablet_manager.cc:1403] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20251212 02:55:25.599159 13526 raft_consensus.cc:359] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:25.599395 13526 raft_consensus.cc:385] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:25.599488 13526 raft_consensus.cc:740] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f54f877d2864fd48526ff4e6b01b9ca, State: Initialized, Role: FOLLOWER
I20251212 02:55:25.599781 13526 consensus_queue.cc:260] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:25.600747 13526 ts_tablet_manager.cc:1434] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251212 02:55:25.602957 13524 ts_tablet_manager.cc:1929] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251212 02:55:25.603137 13524 log.cc:1199] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca: Deleting WAL directory at /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/wal/wals/5b7a8d8cef184c40bb6d5c6a26bc137a
I20251212 02:55:25.603615 13524 ts_tablet_manager.cc:1950] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 5f54f877d2864fd48526ff4e6b01b9ca: Deleting consensus metadata
I20251212 02:55:25.605544 13525 ts_tablet_manager.cc:1929] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251212 02:55:25.605743 13525 log.cc:1199] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75: Deleting WAL directory at /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/wal/wals/5b7a8d8cef184c40bb6d5c6a26bc137a
I20251212 02:55:25.606284 13525 ts_tablet_manager.cc:1950] T 5b7a8d8cef184c40bb6d5c6a26bc137a P 8240d2efd35f443a9515f0db38f09f75: Deleting consensus metadata
I20251212 02:55:25.610103 13528 tablet_bootstrap.cc:492] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf: No bootstrap required, opened a new log
I20251212 02:55:25.610266 13528 ts_tablet_manager.cc:1403] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.007s
I20251212 02:55:25.611233 13528 raft_consensus.cc:359] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:25.611459 13528 raft_consensus.cc:385] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:25.611557 13528 raft_consensus.cc:740] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 480bcf5c1f394464b302900a89ca5bdf, State: Initialized, Role: FOLLOWER
I20251212 02:55:25.611833 13528 consensus_queue.cc:260] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:25.613437 13527 tablet_bootstrap.cc:492] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75: No bootstrap required, opened a new log
I20251212 02:55:25.613602 13527 ts_tablet_manager.cc:1403] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.011s
I20251212 02:55:25.614243 13528 ts_tablet_manager.cc:1434] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20251212 02:55:25.614459 13527 raft_consensus.cc:359] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:25.614658 13527 raft_consensus.cc:385] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:25.614765 13527 raft_consensus.cc:740] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8240d2efd35f443a9515f0db38f09f75, State: Initialized, Role: FOLLOWER
I20251212 02:55:25.616165 13527 consensus_queue.cc:260] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:25.617136 13534 consensus_queue.cc:1048] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 02:55:25.623984 13527 ts_tablet_manager.cc:1434] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75: Time spent starting tablet: real 0.010s user 0.002s sys 0.006s
I20251212 02:55:25.629362 13535 consensus_queue.cc:1048] T f5f45a962b144612bda2cd788c22da2c P 5f54f877d2864fd48526ff4e6b01b9ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 02:55:25.690958 13543 consensus_queue.cc:1048] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 02:55:25.697945 13535 consensus_queue.cc:1048] T 9bf247fbaba4447c9846db800a856dcd P 5f54f877d2864fd48526ff4e6b01b9ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251212 02:55:25.779742 13041 connection.cc:537] server connection from 127.0.0.1:37836 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:25.779780 13188 connection.cc:537] server connection from 127.0.0.1:42936 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:25.779850 13305 connection.cc:537] client connection to 127.12.17.252:44417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:25.780102 13186 connection.cc:537] client connection to 127.12.17.252:44417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:25.780174 12896 connection.cc:537] client connection to 127.12.17.252:44417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:25.780596 13305 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.17.252:44417)
W20251212 02:55:25.780905 13305 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.17.252:44417)
W20251212 02:55:25.782043 13305 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.17.252:44417)
W20251212 02:55:25.783223 13042 connection.cc:537] client connection to 127.12.17.252:44417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:25.783603 12895 connection.cc:537] server connection from 127.0.0.1:37290 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251212 02:55:26.042137 13550 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7823e6e682ac46a09f968e0742506d6e)
I20251212 02:55:26.042397 13550 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:26.044114 13550 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:26.045070 13420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:55:26.046301 13419 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:26.046499 13420 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:26.046703 13420 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:26.047044 13550 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251212 02:55:26.216210 13554 raft_consensus.cc:493] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:26.216439 13554 raft_consensus.cc:515] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:26.218010 13554 leader_election.cc:290] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037), 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:26.224485 13108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef7a386d14cb4d43b9affd638418d0ca" candidate_uuid: "480bcf5c1f394464b302900a89ca5bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8240d2efd35f443a9515f0db38f09f75" is_pre_election: true
I20251212 02:55:26.224717 13108 raft_consensus.cc:2468] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 480bcf5c1f394464b302900a89ca5bdf in term 0.
I20251212 02:55:26.224684 13252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef7a386d14cb4d43b9affd638418d0ca" candidate_uuid: "480bcf5c1f394464b302900a89ca5bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" is_pre_election: true
I20251212 02:55:26.224928 13252 raft_consensus.cc:2468] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 480bcf5c1f394464b302900a89ca5bdf in term 0.
I20251212 02:55:26.225270 12896 leader_election.cc:304] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 480bcf5c1f394464b302900a89ca5bdf, 8240d2efd35f443a9515f0db38f09f75; no voters:
I20251212 02:55:26.225660 13554 raft_consensus.cc:2804] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251212 02:55:26.225786 13554 raft_consensus.cc:493] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 02:55:26.225888 13554 raft_consensus.cc:3060] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:26.227777 13554 raft_consensus.cc:515] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:26.228329 13554 leader_election.cc:290] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [CANDIDATE]: Term 1 election: Requested vote from peers 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037), 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:26.228843 13252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef7a386d14cb4d43b9affd638418d0ca" candidate_uuid: "480bcf5c1f394464b302900a89ca5bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f54f877d2864fd48526ff4e6b01b9ca"
I20251212 02:55:26.228868 13108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef7a386d14cb4d43b9affd638418d0ca" candidate_uuid: "480bcf5c1f394464b302900a89ca5bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8240d2efd35f443a9515f0db38f09f75"
I20251212 02:55:26.229008 13252 raft_consensus.cc:3060] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:26.229041 13108 raft_consensus.cc:3060] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:26.230808 13252 raft_consensus.cc:2468] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 480bcf5c1f394464b302900a89ca5bdf in term 1.
I20251212 02:55:26.230895 13108 raft_consensus.cc:2468] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 480bcf5c1f394464b302900a89ca5bdf in term 1.
I20251212 02:55:26.231230 12897 leader_election.cc:304] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 480bcf5c1f394464b302900a89ca5bdf, 5f54f877d2864fd48526ff4e6b01b9ca; no voters:
I20251212 02:55:26.231524 13554 raft_consensus.cc:2804] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [term 1 FOLLOWER]: Leader election won for term 1
I20251212 02:55:26.231927 13554 raft_consensus.cc:697] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [term 1 LEADER]: Becoming Leader. State: Replica: 480bcf5c1f394464b302900a89ca5bdf, State: Running, Role: LEADER
I20251212 02:55:26.232373 13554 consensus_queue.cc:237] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:26.557389 13563 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7823e6e682ac46a09f968e0742506d6e)
I20251212 02:55:26.557665 13563 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:26.558388 13563 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:26.560554 13419 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:26.560717 13420 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:26.560832 13420 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:26.561072 13563 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251212 02:55:26.994961 13570 consensus_queue.cc:1048] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 02:55:27.003115 13574 consensus_queue.cc:1048] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 02:55:27.331889 13580 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7823e6e682ac46a09f968e0742506d6e)
I20251212 02:55:27.332161 13580 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:27.332989 13580 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:27.335191 13419 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:27.335373 13420 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:27.335516 13420 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:27.335810 13580 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251212 02:55:28.123237 12359 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
/tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.254:40649
--webserver_interface=127.12.17.254
--webserver_port=35303
--builtin_ntp_servers=127.12.17.212:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251212 02:55:28.455507 13595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251212 02:55:28.455828 13595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251212 02:55:28.455955 13595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251212 02:55:28.464221 13595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251212 02:55:28.464318 13595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251212 02:55:28.464381 13595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251212 02:55:28.464457 13595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251212 02:55:28.474342 13595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.17.212:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.254:40649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.17.254
--webserver_port=35303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 12 Dec 2025 02:43:20 UTC on 24a791456cd2
build id 9472
ASAN enabled
I20251212 02:55:28.476172 13595 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 02:55:28.478193 13595 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 02:55:28.486691 13602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:28.486831 13601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:28.488646 13604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:28.489264 13595 server_base.cc:1047] running on GCE node
I20251212 02:55:28.490216 13595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251212 02:55:28.495610 13595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251212 02:55:28.496927 13595 hybrid_clock.cc:648] HybridClock initialized: now 1765508128496802 us; error 122 us; skew 500 ppm
I20251212 02:55:28.497431 13595 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251212 02:55:28.499909 13595 webserver.cc:492] Webserver started at http://127.12.17.254:35303/ using document root <none> and password file <none>
I20251212 02:55:28.500643 13595 fs_manager.cc:362] Metadata directory not provided
I20251212 02:55:28.500749 13595 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 02:55:28.506172 13595 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20251212 02:55:28.509294 13614 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:28.510766 13595 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251212 02:55:28.510936 13595 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
uuid: "62ed31a99dac451981f4562901f11e87"
format_stamp: "Formatted at 2025-12-12 02:55:20 on dist-test-slave-8cz7"
I20251212 02:55:28.511917 13595 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 02:55:28.537969 13595 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 02:55:28.538863 13595 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 02:55:28.539134 13595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 02:55:28.559808 13595 rpc_server.cc:307] RPC server started. Bound to: 127.12.17.254:40649
I20251212 02:55:28.559830 13667 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.17.254:40649 every 8 connection(s)
I20251212 02:55:28.561472 13595 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/info.pb
I20251212 02:55:28.562733 12359 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu as pid 13595
I20251212 02:55:28.570124 13668 sys_catalog.cc:263] Verifying existing consensus state
I20251212 02:55:28.574743 13668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Bootstrap starting.
I20251212 02:55:28.586956 13668 log.cc:826] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Log is configured to *not* fsync() on all Append() calls
I20251212 02:55:28.599746 13668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=3 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251212 02:55:28.600289 13668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Bootstrap complete.
I20251212 02:55:28.605608 13668 raft_consensus.cc:359] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:28.606302 13668 raft_consensus.cc:740] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ed31a99dac451981f4562901f11e87, State: Initialized, Role: FOLLOWER
I20251212 02:55:28.606935 13668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:28.607746 13673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:28.607965 13673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:28.608654 13668 sys_catalog.cc:565] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: configured and running, proceeding with master startup.
I20251212 02:55:28.621973 13687 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:28.622188 13687 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: loading cluster ID for follower catalog manager: success
I20251212 02:55:28.626322 13687 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: acquiring CA information for follower catalog manager: success
I20251212 02:55:28.629127 13687 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251212 02:55:28.644577 13595 master_runner.cc:186] Error getting master registration for 127.12.17.252:44417: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111),
I20251212 02:55:28.650135 13595 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:55:28.832825 13693 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7823e6e682ac46a09f968e0742506d6e)
I20251212 02:55:28.833030 13693 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:28.833948 13693 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:28.838267 13420 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:28.842914 13643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad856a8b02154355b5da573e8fffcb16" candidate_term: 5 candidate_status { last_received { term: 4 index: 15 } } ignore_live_leader: false dest_uuid: "62ed31a99dac451981f4562901f11e87" is_pre_election: true
I20251212 02:55:28.843523 13643 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad856a8b02154355b5da573e8fffcb16 in term 3.
I20251212 02:55:28.844185 13419 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16; no voters: 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:28.844507 13693 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251212 02:55:28.844641 13693 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7823e6e682ac46a09f968e0742506d6e)
I20251212 02:55:28.844741 13693 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 4 FOLLOWER]: Advancing to term 5
I20251212 02:55:28.846755 13693 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:28.847352 13693 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 election: Requested vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
I20251212 02:55:28.847855 13643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad856a8b02154355b5da573e8fffcb16" candidate_term: 5 candidate_status { last_received { term: 4 index: 15 } } ignore_live_leader: false dest_uuid: "62ed31a99dac451981f4562901f11e87"
I20251212 02:55:28.848096 13643 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 3 FOLLOWER]: Advancing to term 5
W20251212 02:55:28.848924 13420 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:28.852289 13643 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad856a8b02154355b5da573e8fffcb16 in term 5.
I20251212 02:55:28.852725 13419 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16; no voters: 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:28.853018 13693 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 5 FOLLOWER]: Leader election won for term 5
I20251212 02:55:28.853508 13693 raft_consensus.cc:697] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 5 LEADER]: Becoming Leader. State: Replica: ad856a8b02154355b5da573e8fffcb16, State: Running, Role: LEADER
I20251212 02:55:28.854056 13693 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 4.15, Last appended by leader: 15, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:28.855975 13697 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad856a8b02154355b5da573e8fffcb16. Latest consensus state: current_term: 5 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:28.856272 13697 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: LEADER
I20251212 02:55:28.856983 13699 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251212 02:55:28.859403 13699 catalog_manager.cc:679] Loaded metadata for table default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c]
I20251212 02:55:28.859915 13699 catalog_manager.cc:679] Loaded metadata for table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]
I20251212 02:55:28.860272 13699 catalog_manager.cc:679] Loaded metadata for table default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e]
I20251212 02:55:28.862092 13699 tablet_loader.cc:96] loaded metadata for tablet 9bf247fbaba4447c9846db800a856dcd (table default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e])
I20251212 02:55:28.862437 13699 tablet_loader.cc:96] loaded metadata for tablet ef7a386d14cb4d43b9affd638418d0ca (table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803])
I20251212 02:55:28.862808 13699 tablet_loader.cc:96] loaded metadata for tablet f5f45a962b144612bda2cd788c22da2c (table default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c])
I20251212 02:55:28.863057 13699 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251212 02:55:28.863914 13699 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:28.863986 13699 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251212 02:55:28.865123 13699 catalog_manager.cc:1514] Loading token signing keys...
I20251212 02:55:28.865938 13699 catalog_manager.cc:6038] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Loaded TSK: 0
I20251212 02:55:28.866787 13699 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251212 02:55:28.867614 13699 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251212 02:55:28.868427 13699 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251212 02:55:28.889247 13434 master_service.cc:954] ReplaceTablet: received request to replace tablet f5f45a962b144612bda2cd788c22da2c from {username='slave'} at 127.0.0.1:54214
I20251212 02:55:28.890264 13433 master_service.cc:954] ReplaceTablet: received request to replace tablet 9bf247fbaba4447c9846db800a856dcd from {username='slave'} at 127.0.0.1:54214
I20251212 02:55:28.892335 13431 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54214:
name: "default.Table_4f27d7a3eec743a587a57dd44c7c649c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:28.892653 13432 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54214:
name: "default.Table_b815b4e2b12a41a9a3e0d97ab076f666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:28.893978 13426 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54214:
name: "default.Table_fc25d2c6f4ad42958441b5b6c7eab15c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:28.894676 13428 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54214:
name: "default.Table_397fb5c3eafe4e49b3667efcf7811f7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:28.895310 13430 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54214:
name: "default.Table_a63f5d536bfd491fb1d666840afc41a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:28.897868 13427 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54214:
name: "default.Table_2bed983ea7fa4fd39e17cffd5a24225f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:28.894520 13429 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54214:
name: "default.Table_70d527cb5d6a46c0b3a5a994b5db9375"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:28.899531 13435 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54214:
name: "default.Table_9dd8cc0b642941cfadbb98ab83b7f175"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251212 02:55:28.900280 13488 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1765508123, eventType=CREATE_TABLE, dbName=default, tableName=table_f6ccb5471f934cf0a9ff7e662f65b997, message={"server":"","servicePrincipal":"","db":"default","table":"table_f6ccb5471f934cf0a9ff7e662f65b997","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f6ccb5471f934cf0a9ff7e662f65b997\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765508123},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/warehouse/table_f6ccb5471f934cf0a9ff7e662f65b997\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f6ccb5471f934cf0a9ff7e662f65b997\",\"kudu.table_id\":\"45586ce12b354c4cac23e4a22aafe67c\",\"transient_lastDdlTime\":\"1765508123\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cbec40b29e204890941f676300eb9cd8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765508123,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765508123, eventType=CREATE_TABLE, dbName=default, tableName=table_a63f5d536bfd491fb1d666840afc41a4, message={"server":"","servicePrincipal":"","db":"default","table":"table_a63f5d536bfd491fb1d666840afc41a4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a63f5d536bfd491fb1d666840afc41a4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765508123},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/warehouse/table_a63f5d536bfd491fb1d666840afc41a4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a63f5d536bfd491fb1d666840afc41a4\",\"kudu.table_id\":\"6f9b849174f44f8b94870d02244aeba0\",\"transient_lastDdlTime\":\"1765508123\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cbec40b29e204890941f676300eb9cd8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765508123,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f010ac5fdcd google::LogMessage::Fail() at ??:0
I20251212 02:55:28.903942 13643 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Refusing update from remote peer ad856a8b02154355b5da573e8fffcb16: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
@ 0x7f010ac63b93 google::LogMessage::SendToLog() at ??:0
W20251212 02:55:28.905354 13420 consensus_peers.cc:597] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 -> Peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Couldn't send request to peer 7823e6e682ac46a09f968e0742506d6e. Status: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251212 02:55:28.905426 13697 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 12, Time since last communication: 0.000s
@ 0x7f010ac5f7cc google::LogMessage::Flush() at ??:0
@ 0x7f010ac5fa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f0121626aa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251212 02:55:28.923386 13673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad856a8b02154355b5da573e8fffcb16. Latest consensus state: current_term: 5 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:28.923671 13673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:28.923547 13704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:28.923949 13704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f0121625351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251212 02:55:28.927385 13704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:28.927639 13704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f012162cfcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251212 02:55:28.940038 13702 mvcc.cc:204] Tried to move back new op lower bound from 7231521295988416512 to 7231521281719238656. Current Snapshot: MvccSnapshot[applied={T|T < 7231521295988416512}]
I20251212 02:55:28.942893 13434 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Sending DeleteTablet for 3 replicas of tablet f5f45a962b144612bda2cd788c22da2c
I20251212 02:55:28.945071 13434 catalog_manager.cc:6724] ReplaceTablet: tablet f5f45a962b144612bda2cd788c22da2c deleted and replaced by tablet 182044220c484c968b45914e5d037e64
@ 0x7f012162ce49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7f010d9d43f9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0110fdf6db start_thread at ??:0
@ 0x7f0109b1071f clone at ??:0
I20251212 02:55:29.006680 13008 heartbeater.cc:344] Connected to a master server at 127.12.17.254:40649
I20251212 02:55:29.006886 13008 heartbeater.cc:499] Master 127.12.17.254:40649 was elected leader, sending a full tablet report...
I20251212 02:55:29.012045 13632 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" instance_seqno: 1765508122427264) as {username='slave'} at 127.12.17.193:39405; Asking this server to re-register.
I20251212 02:55:29.014389 13008 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:29.026309 13632 ts_manager.cc:194] Registered new tserver with Master: 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431)
I20251212 02:55:29.030543 13298 heartbeater.cc:344] Connected to a master server at 127.12.17.254:40649
I20251212 02:55:29.030767 13298 heartbeater.cc:499] Master 127.12.17.254:40649 was elected leader, sending a full tablet report...
I20251212 02:55:29.032968 13633 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" instance_seqno: 1765508123473667) as {username='slave'} at 127.12.17.195:48407; Asking this server to re-register.
I20251212 02:55:29.033778 13298 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:29.035635 13632 ts_manager.cc:194] Registered new tserver with Master: 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:29.040721 13154 heartbeater.cc:344] Connected to a master server at 127.12.17.254:40649
I20251212 02:55:29.040997 13154 heartbeater.cc:499] Master 127.12.17.254:40649 was elected leader, sending a full tablet report...
I20251212 02:55:29.043251 13632 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" instance_seqno: 1765508122990947) as {username='slave'} at 127.12.17.194:59801; Asking this server to re-register.
I20251212 02:55:29.044196 13154 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:29.046192 13632 ts_manager.cc:194] Registered new tserver with Master: 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
W20251212 02:55:29.185547 13712 negotiation.cc:337] Failed RPC negotiation. Trace:
1212 02:55:28.945543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.12.17.194:43037)
1212 02:55:28.964565 (+ 19022us) server_negotiation.cc:207] Beginning negotiation
1212 02:55:28.964574 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1212 02:55:28.964609 (+ 35us) server_negotiation.cc:408] Connection header received
1212 02:55:28.964708 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1212 02:55:28.964718 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1212 02:55:28.964842 (+ 124us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1212 02:55:28.964976 (+ 134us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1212 02:55:29.185055 (+220079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43664: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":18836,"thread_start_us":19631,"threads_started":1}
W20251212 02:55:29.185503 13304 connection.cc:537] client connection to 127.12.17.253:46623 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:29.185606 13185 connection.cc:537] client connection to 127.12.17.253:46623 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:29.186128 13715 negotiation.cc:337] Failed RPC negotiation. Trace:
1212 02:55:28.947283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.12.17.193:40431)
1212 02:55:28.954373 (+ 7090us) server_negotiation.cc:207] Beginning negotiation
1212 02:55:28.954382 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1212 02:55:28.960724 (+ 6342us) server_negotiation.cc:408] Connection header received
1212 02:55:28.960828 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1212 02:55:28.960837 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1212 02:55:28.960934 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1212 02:55:28.961871 (+ 937us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1212 02:55:28.962078 (+ 207us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1212 02:55:28.963755 (+ 1677us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1212 02:55:29.185054 (+221299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41482: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":6951,"thread_start_us":6843,"threads_started":1}
W20251212 02:55:29.186226 13304 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.17.253:46623)
W20251212 02:55:29.186741 13304 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.17.253:46623)
W20251212 02:55:29.187007 13041 connection.cc:537] client connection to 127.12.17.253:46623 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:29.187530 12895 connection.cc:537] client connection to 127.12.17.253:46623 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:29.187951 13304 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.17.253:46623)
I20251212 02:55:29.427841 13704 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ad856a8b02154355b5da573e8fffcb16)
I20251212 02:55:29.428095 13704 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:29.430543 13704 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:29.431411 13617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:55:29.432682 13618 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:55:29.432864 13617 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:29.433012 13617 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:29.433442 13704 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251212 02:55:30.130689 13734 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ad856a8b02154355b5da573e8fffcb16)
I20251212 02:55:30.130901 13734 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:30.131664 13734 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:30.133634 13617 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:55:30.133808 13618 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:30.133956 13618 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:30.134183 13734 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251212 02:55:31.062278 13741 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ad856a8b02154355b5da573e8fffcb16)
I20251212 02:55:31.062470 13741 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:31.063103 13741 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:31.065162 13618 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:55:31.065418 13617 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:31.065536 13617 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:31.065781 13741 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251212 02:55:31.721267 12359 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
/tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.253:46623
--webserver_interface=127.12.17.253
--webserver_port=36499
--builtin_ntp_servers=127.12.17.212:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251212 02:55:32.061826 13747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251212 02:55:32.062172 13747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251212 02:55:32.062294 13747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251212 02:55:32.070729 13747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251212 02:55:32.070833 13747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251212 02:55:32.070888 13747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251212 02:55:32.070928 13747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251212 02:55:32.080876 13747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.17.212:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.253:46623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.17.253
--webserver_port=36499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 12 Dec 2025 02:43:20 UTC on 24a791456cd2
build id 9472
ASAN enabled
I20251212 02:55:32.082598 13747 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 02:55:32.084429 13747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251212 02:55:32.092105 13754 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ad856a8b02154355b5da573e8fffcb16)
I20251212 02:55:32.092453 13754 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
W20251212 02:55:32.092423 13752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:32.092507 13753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:32.093432 13756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:32.093622 13754 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
I20251212 02:55:32.095316 13747 server_base.cc:1047] running on GCE node
W20251212 02:55:32.095626 13618 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:55:32.095856 13617 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:32.095965 13617 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:32.096197 13754 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251212 02:55:32.096249 13747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251212 02:55:32.097733 13747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251212 02:55:32.098929 13747 hybrid_clock.cc:648] HybridClock initialized: now 1765508132098878 us; error 37 us; skew 500 ppm
I20251212 02:55:32.099280 13747 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251212 02:55:32.101536 13747 webserver.cc:492] Webserver started at http://127.12.17.253:36499/ using document root <none> and password file <none>
I20251212 02:55:32.102299 13747 fs_manager.cc:362] Metadata directory not provided
I20251212 02:55:32.102394 13747 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 02:55:32.107842 13747 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20251212 02:55:32.110956 13765 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:32.112541 13747 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251212 02:55:32.112712 13747 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data,/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal
uuid: "ad856a8b02154355b5da573e8fffcb16"
format_stamp: "Formatted at 2025-12-12 02:55:20 on dist-test-slave-8cz7"
I20251212 02:55:32.113493 13747 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 02:55:32.138927 13747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 02:55:32.139909 13747 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 02:55:32.140205 13747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 02:55:32.160730 13747 rpc_server.cc:307] RPC server started. Bound to: 127.12.17.253:46623
I20251212 02:55:32.160773 13816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.17.253:46623 every 8 connection(s)
I20251212 02:55:32.162421 13747 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-1/data/info.pb
I20251212 02:55:32.169214 13817 sys_catalog.cc:263] Verifying existing consensus state
I20251212 02:55:32.170264 12359 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu as pid 13747
I20251212 02:55:32.172886 13817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Bootstrap starting.
I20251212 02:55:32.183039 13817 log.cc:826] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Log is configured to *not* fsync() on all Append() calls
I20251212 02:55:32.202863 13299 heartbeater.cc:344] Connected to a master server at 127.12.17.253:46623
I20251212 02:55:32.204808 13817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=17 ignored=0} inserts{seen=11 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251212 02:55:32.209904 13817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Bootstrap complete.
I20251212 02:55:32.212350 13155 heartbeater.cc:344] Connected to a master server at 127.12.17.253:46623
I20251212 02:55:32.213486 13009 heartbeater.cc:344] Connected to a master server at 127.12.17.253:46623
I20251212 02:55:32.217775 13817 raft_consensus.cc:359] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 5 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:32.220434 13817 raft_consensus.cc:740] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad856a8b02154355b5da573e8fffcb16, State: Initialized, Role: FOLLOWER
I20251212 02:55:32.221055 13817 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 5.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:32.221778 13825 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:32.222087 13825 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:32.222787 13817 sys_catalog.cc:565] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: configured and running, proceeding with master startup.
I20251212 02:55:32.248853 13839 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:32.248961 13839 catalog_manager.cc:1562] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: loading cluster ID for follower catalog manager: success
I20251212 02:55:32.252806 13839 catalog_manager.cc:1584] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: acquiring CA information for follower catalog manager: success
I20251212 02:55:32.255445 13839 catalog_manager.cc:1612] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251212 02:55:32.269361 13747 master_runner.cc:186] Error getting master registration for 127.12.17.252:44417: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111),
I20251212 02:55:32.272773 13747 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:55:32.639818 13844 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ad856a8b02154355b5da573e8fffcb16)
I20251212 02:55:32.640020 13844 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:32.640684 13844 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:32.642681 13618 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:32.646848 13792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ed31a99dac451981f4562901f11e87" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "ad856a8b02154355b5da573e8fffcb16" is_pre_election: true
I20251212 02:55:32.647356 13792 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62ed31a99dac451981f4562901f11e87 in term 5.
I20251212 02:55:32.647821 13617 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16; no voters: 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:32.648145 13844 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251212 02:55:32.648264 13844 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ad856a8b02154355b5da573e8fffcb16)
I20251212 02:55:32.648331 13844 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 5 FOLLOWER]: Advancing to term 6
I20251212 02:55:32.650262 13844 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:32.650828 13844 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 election: Requested vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
I20251212 02:55:32.651264 13792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ed31a99dac451981f4562901f11e87" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "ad856a8b02154355b5da573e8fffcb16"
I20251212 02:55:32.651510 13792 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 5 FOLLOWER]: Advancing to term 6
W20251212 02:55:32.652136 13618 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:32.655038 13792 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62ed31a99dac451981f4562901f11e87 in term 6.
I20251212 02:55:32.655378 13617 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16; no voters: 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:32.655628 13844 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 6 FOLLOWER]: Leader election won for term 6
I20251212 02:55:32.656003 13844 raft_consensus.cc:697] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 6 LEADER]: Becoming Leader. State: Replica: 62ed31a99dac451981f4562901f11e87, State: Running, Role: LEADER
I20251212 02:55:32.656389 13844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 5.18, Last appended by leader: 18, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:32.657895 13848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ed31a99dac451981f4562901f11e87. Latest consensus state: current_term: 6 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:32.658169 13848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: LEADER
I20251212 02:55:32.658793 13850 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251212 02:55:32.661068 13850 catalog_manager.cc:679] Loaded metadata for table default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c]
I20251212 02:55:32.661543 13850 catalog_manager.cc:679] Loaded metadata for table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]
I20251212 02:55:32.661891 13850 catalog_manager.cc:679] Loaded metadata for table default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e]
I20251212 02:55:32.663343 13850 tablet_loader.cc:96] loaded metadata for tablet 182044220c484c968b45914e5d037e64 (table default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c])
I20251212 02:55:32.663831 13850 tablet_loader.cc:96] loaded metadata for tablet 702cfe1a3e75438ba932d34354ffe0b5 (table default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e])
I20251212 02:55:32.664306 13850 tablet_loader.cc:96] loaded metadata for tablet ef7a386d14cb4d43b9affd638418d0ca (table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803])
I20251212 02:55:32.664705 13850 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251212 02:55:32.665557 13850 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:32.665638 13850 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251212 02:55:32.666730 13850 catalog_manager.cc:1514] Loading token signing keys...
I20251212 02:55:32.667507 13850 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Loaded TSK: 0
I20251212 02:55:32.668316 13850 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251212 02:55:32.669072 13850 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251212 02:55:32.669875 13850 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251212 02:55:32.703223 13633 master_service.cc:954] ReplaceTablet: received request to replace tablet f5f45a962b144612bda2cd788c22da2c from {username='slave'} at 127.0.0.1:38454
I20251212 02:55:32.703313 13632 master_service.cc:954] ReplaceTablet: received request to replace tablet 9bf247fbaba4447c9846db800a856dcd from {username='slave'} at 127.0.0.1:38454
I20251212 02:55:32.704020 13631 master_service.cc:954] ReplaceTablet: received request to replace tablet 9bf247fbaba4447c9846db800a856dcd from {username='slave'} at 127.0.0.1:38454
I20251212 02:55:32.704188 13630 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38454:
name: "default.Table_a63f5d536bfd491fb1d666840afc41a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:32.705257 13633 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38454:
name: "default.Table_4f27d7a3eec743a587a57dd44c7c649c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:32.705499 13632 master_service.cc:954] ReplaceTablet: received request to replace tablet 9bf247fbaba4447c9846db800a856dcd from {username='slave'} at 127.0.0.1:38454
I20251212 02:55:32.705781 13629 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38454:
name: "default.Table_9dd8cc0b642941cfadbb98ab83b7f175"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:32.707566 13627 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38454:
name: "default.Table_397fb5c3eafe4e49b3667efcf7811f7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:32.708227 13782 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" instance_seqno: 1765508123473667) as {username='slave'} at 127.12.17.195:43915; Asking this server to re-register.
I20251212 02:55:32.708590 13632 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38454:
name: "default.Table_2bed983ea7fa4fd39e17cffd5a24225f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:32.709004 13299 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:32.709288 13626 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38454:
name: "default.Table_fc25d2c6f4ad42958441b5b6c7eab15c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:32.709501 13625 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38454:
name: "default.Table_aeb57448bc884b4ca11ecdd045932bc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:32.711284 13624 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38454:
name: "default.Table_2a0b5225731647628e4f9ecd5a0ffb2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:32.711609 13781 ts_manager.cc:194] Registered new tserver with Master: 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:32.712029 13628 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38454:
name: "default.Table_70d527cb5d6a46c0b3a5a994b5db9375"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:32.712107 13631 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38454:
name: "default.Table_b815b4e2b12a41a9a3e0d97ab076f666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251212 02:55:32.715729 13686 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1765508123, eventType=CREATE_TABLE, dbName=default, tableName=table_f6ccb5471f934cf0a9ff7e662f65b997, message={"server":"","servicePrincipal":"","db":"default","table":"table_f6ccb5471f934cf0a9ff7e662f65b997","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f6ccb5471f934cf0a9ff7e662f65b997\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765508123},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/warehouse/table_f6ccb5471f934cf0a9ff7e662f65b997\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f6ccb5471f934cf0a9ff7e662f65b997\",\"kudu.table_id\":\"45586ce12b354c4cac23e4a22aafe67c\",\"transient_lastDdlTime\":\"1765508123\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cbec40b29e204890941f676300eb9cd8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765508123,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765508123, eventType=CREATE_TABLE, dbName=default, tableName=table_a63f5d536bfd491fb1d666840afc41a4, message={"server":"","servicePrincipal":"","db":"default","table":"table_a63f5d536bfd491fb1d666840afc41a4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a63f5d536bfd491fb1d666840afc41a4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765508123},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/warehouse/table_a63f5d536bfd491fb1d666840afc41a4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a63f5d536bfd491fb1d666840afc41a4\",\"kudu.table_id\":\"6f9b849174f44f8b94870d02244aeba0\",\"transient_lastDdlTime\":\"1765508123\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cbec40b29e204890941f676300eb9cd8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765508123,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251212 02:55:32.716377 13782 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" instance_seqno: 1765508122427264) as {username='slave'} at 127.12.17.193:52303; Asking this server to re-register.
I20251212 02:55:32.716950 13009 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:32.717634 13781 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" instance_seqno: 1765508122990947) as {username='slave'} at 127.12.17.194:37773; Asking this server to re-register.
I20251212 02:55:32.718312 13155 heartbeater.cc:461] Registering TS with master...
@ 0x7f733ade1dcd google::LogMessage::Fail() at ??:0
I20251212 02:55:32.719311 13782 ts_manager.cc:194] Registered new tserver with Master: 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431)
I20251212 02:55:32.719848 13781 ts_manager.cc:194] Registered new tserver with Master: 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
@ 0x7f733ade5b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f733ade17cc google::LogMessage::Flush() at ??:0
@ 0x7f733ade1a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f73517a8aa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f73517a7351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f73517aefcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f73517aee49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7f733db563f9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f73411616db start_thread at ??:0
@ 0x7f7339c9271f clone at ??:0
W20251212 02:55:32.916976 13304 connection.cc:537] client connection to 127.12.17.254:40649 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:32.917524 13304 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.17.254:40649)
W20251212 02:55:32.918051 13304 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.17.254:40649)
W20251212 02:55:32.918264 13304 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.17.254:40649)
I20251212 02:55:33.364405 13867 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:33.364609 13867 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:33.366496 13867 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:33.367372 13766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:33.368477 13767 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:55:33.368678 13766 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
I20251212 02:55:33.368806 13766 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:33.369107 13867 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251212 02:55:33.957098 13873 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:33.957330 13873 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:33.957931 13873 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:33.960078 13766 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:33.960237 13767 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:33.960353 13767 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:33.960569 13873 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251212 02:55:34.646224 13877 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:34.646441 13877 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:34.647159 13877 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:34.649889 13767 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:55:34.650079 13766 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
I20251212 02:55:34.650202 13766 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:34.650441 13877 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251212 02:55:35.035871 12359 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
/tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.254:40649
--webserver_interface=127.12.17.254
--webserver_port=35303
--builtin_ntp_servers=127.12.17.212:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251212 02:55:35.365028 13882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251212 02:55:35.365424 13882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251212 02:55:35.365545 13882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251212 02:55:35.373723 13882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251212 02:55:35.373817 13882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251212 02:55:35.373901 13882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251212 02:55:35.373968 13882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251212 02:55:35.383752 13882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.17.212:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.254:40649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.17.254
--webserver_port=35303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 12 Dec 2025 02:43:20 UTC on 24a791456cd2
build id 9472
ASAN enabled
I20251212 02:55:35.385463 13882 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 02:55:35.387225 13882 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 02:55:35.395277 13887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:35.395305 13888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:35.395866 13890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:35.396433 13882 server_base.cc:1047] running on GCE node
I20251212 02:55:35.397125 13882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251212 02:55:35.398474 13882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251212 02:55:35.399669 13882 hybrid_clock.cc:648] HybridClock initialized: now 1765508135399625 us; error 33 us; skew 500 ppm
I20251212 02:55:35.399977 13882 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251212 02:55:35.402141 13882 webserver.cc:492] Webserver started at http://127.12.17.254:35303/ using document root <none> and password file <none>
I20251212 02:55:35.402830 13882 fs_manager.cc:362] Metadata directory not provided
I20251212 02:55:35.402935 13882 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 02:55:35.408217 13882 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20251212 02:55:35.411167 13896 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:35.412465 13882 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251212 02:55:35.412623 13882 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
uuid: "62ed31a99dac451981f4562901f11e87"
format_stamp: "Formatted at 2025-12-12 02:55:20 on dist-test-slave-8cz7"
I20251212 02:55:35.413403 13882 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 02:55:35.433632 13882 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 02:55:35.434473 13882 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 02:55:35.434736 13882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 02:55:35.455974 13882 rpc_server.cc:307] RPC server started. Bound to: 127.12.17.254:40649
I20251212 02:55:35.455996 13947 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.17.254:40649 every 8 connection(s)
I20251212 02:55:35.457587 13882 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/info.pb
I20251212 02:55:35.464449 12359 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu as pid 13882
I20251212 02:55:35.464597 13948 sys_catalog.cc:263] Verifying existing consensus state
I20251212 02:55:35.468654 13948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Bootstrap starting.
I20251212 02:55:35.477689 13948 log.cc:826] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Log is configured to *not* fsync() on all Append() calls
I20251212 02:55:35.497275 13948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=18 ignored=0} inserts{seen=12 ignored=0} mutations{seen=7 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251212 02:55:35.497880 13948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Bootstrap complete.
I20251212 02:55:35.503190 13948 raft_consensus.cc:359] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:35.504099 13948 raft_consensus.cc:740] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ed31a99dac451981f4562901f11e87, State: Initialized, Role: FOLLOWER
I20251212 02:55:35.504690 13948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 6.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:35.505550 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:35.505836 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:35.506618 13948 sys_catalog.cc:565] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: configured and running, proceeding with master startup.
I20251212 02:55:35.529573 13966 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:35.531610 13966 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: loading cluster ID for follower catalog manager: success
I20251212 02:55:35.536172 13966 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: acquiring CA information for follower catalog manager: success
I20251212 02:55:35.539157 13966 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251212 02:55:35.545101 13971 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:35.545707 13971 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:35.546548 13971 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:35.553314 13767 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:35.551888 13882 master_runner.cc:186] Error getting master registration for 127.12.17.252:44417: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111),
I20251212 02:55:35.559235 13882 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:55:35.562158 13923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad856a8b02154355b5da573e8fffcb16" candidate_term: 7 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "62ed31a99dac451981f4562901f11e87" is_pre_election: true
I20251212 02:55:35.563010 13923 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ad856a8b02154355b5da573e8fffcb16 for term 7 because replica has last-logged OpId of term: 6 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 18.
I20251212 02:55:35.563329 13008 heartbeater.cc:344] Connected to a master server at 127.12.17.254:40649
I20251212 02:55:35.564661 13766 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:35.565963 13971 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251212 02:55:35.566129 13298 heartbeater.cc:344] Connected to a master server at 127.12.17.254:40649
I20251212 02:55:35.566345 13913 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" instance_seqno: 1765508122427264) as {username='slave'} at 127.12.17.193:39969; Asking this server to re-register.
I20251212 02:55:35.567283 13008 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:35.569518 13912 ts_manager.cc:194] Registered new tserver with Master: 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431)
I20251212 02:55:35.570760 13911 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" instance_seqno: 1765508123473667) as {username='slave'} at 127.12.17.195:60573; Asking this server to re-register.
I20251212 02:55:35.571501 13298 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:35.572945 13911 ts_manager.cc:194] Registered new tserver with Master: 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:35.575242 13154 heartbeater.cc:344] Connected to a master server at 127.12.17.254:40649
I20251212 02:55:35.576234 13911 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" instance_seqno: 1765508122990947) as {username='slave'} at 127.12.17.194:47099; Asking this server to re-register.
I20251212 02:55:35.576689 13154 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:35.577693 13911 ts_manager.cc:194] Registered new tserver with Master: 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
I20251212 02:55:35.948635 13952 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:35.948917 13952 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:35.950771 13952 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
I20251212 02:55:35.951524 13792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ed31a99dac451981f4562901f11e87" candidate_term: 7 candidate_status { last_received { term: 6 index: 19 } } ignore_live_leader: false dest_uuid: "ad856a8b02154355b5da573e8fffcb16" is_pre_election: true
W20251212 02:55:35.951610 13898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:35.951764 13792 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62ed31a99dac451981f4562901f11e87 in term 6.
I20251212 02:55:35.952356 13897 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16; no voters:
I20251212 02:55:35.952697 13952 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251212 02:55:35.952834 13952 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 02:55:35.952947 13952 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 6 FOLLOWER]: Advancing to term 7
W20251212 02:55:35.953109 13898 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:35.956516 13952 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:35.957069 13952 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 7 election: Requested vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
I20251212 02:55:35.957485 13792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ed31a99dac451981f4562901f11e87" candidate_term: 7 candidate_status { last_received { term: 6 index: 19 } } ignore_live_leader: false dest_uuid: "ad856a8b02154355b5da573e8fffcb16"
I20251212 02:55:35.957670 13792 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 6 FOLLOWER]: Advancing to term 7
W20251212 02:55:35.958295 13898 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:35.959513 13792 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62ed31a99dac451981f4562901f11e87 in term 7.
I20251212 02:55:35.959867 13897 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16; no voters: 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:35.960121 13952 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 7 FOLLOWER]: Leader election won for term 7
I20251212 02:55:35.960515 13952 raft_consensus.cc:697] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 7 LEADER]: Becoming Leader. State: Replica: 62ed31a99dac451981f4562901f11e87, State: Running, Role: LEADER
I20251212 02:55:35.960963 13952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 6.19, Last appended by leader: 19, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:35.963639 13980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ed31a99dac451981f4562901f11e87. Latest consensus state: current_term: 7 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:35.963881 13980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: LEADER
I20251212 02:55:35.964505 13982 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251212 02:55:35.966864 13982 catalog_manager.cc:679] Loaded metadata for table default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c]
I20251212 02:55:35.967312 13982 catalog_manager.cc:679] Loaded metadata for table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]
I20251212 02:55:35.967644 13982 catalog_manager.cc:679] Loaded metadata for table default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e]
I20251212 02:55:35.969098 13982 tablet_loader.cc:96] loaded metadata for tablet 182044220c484c968b45914e5d037e64 (table default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c])
I20251212 02:55:35.969606 13982 tablet_loader.cc:96] loaded metadata for tablet 702cfe1a3e75438ba932d34354ffe0b5 (table default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e])
I20251212 02:55:35.970060 13982 tablet_loader.cc:96] loaded metadata for tablet ef7a386d14cb4d43b9affd638418d0ca (table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803])
I20251212 02:55:35.970444 13982 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251212 02:55:35.971282 13982 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:35.971349 13982 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251212 02:55:35.972543 13982 catalog_manager.cc:1514] Loading token signing keys...
I20251212 02:55:35.973402 13982 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Loaded TSK: 0
I20251212 02:55:35.974313 13982 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251212 02:55:35.975061 13982 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251212 02:55:35.975771 13982 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251212 02:55:36.031924 13911 master_service.cc:954] ReplaceTablet: received request to replace tablet 9bf247fbaba4447c9846db800a856dcd from {username='slave'} at 127.0.0.1:39144
I20251212 02:55:36.032183 13913 master_service.cc:954] ReplaceTablet: received request to replace tablet 9bf247fbaba4447c9846db800a856dcd from {username='slave'} at 127.0.0.1:39144
I20251212 02:55:36.031924 13912 master_service.cc:954] ReplaceTablet: received request to replace tablet ef7a386d14cb4d43b9affd638418d0ca from {username='slave'} at 127.0.0.1:39144
I20251212 02:55:36.032274 13910 master_service.cc:954] ReplaceTablet: received request to replace tablet 9bf247fbaba4447c9846db800a856dcd from {username='slave'} at 127.0.0.1:39144
I20251212 02:55:36.033087 13908 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39144:
table { table_name: "default.Table_b36fce53138b45fc9fb1a2d7ea2afae7" } new_table_name: "default.Table_2f6246f54e0e491287fe6061611a2ca9" modify_external_catalogs: true
I20251212 02:55:36.033445 13909 master_service.cc:954] ReplaceTablet: received request to replace tablet 9bf247fbaba4447c9846db800a856dcd from {username='slave'} at 127.0.0.1:39144
I20251212 02:55:36.033412 13907 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39144:
name: "default.Table_b815b4e2b12a41a9a3e0d97ab076f666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:36.034550 13913 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39144:
name: "default.Table_a63f5d536bfd491fb1d666840afc41a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:36.034585 13911 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39144:
name: "default.Table_4f27d7a3eec743a587a57dd44c7c649c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:36.034732 13909 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39144:
name: "default.Table_397fb5c3eafe4e49b3667efcf7811f7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:36.036063 13910 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39144:
name: "default.Table_fc25d2c6f4ad42958441b5b6c7eab15c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:36.035843 13905 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39144:
name: "default.Table_70d527cb5d6a46c0b3a5a994b5db9375"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:36.036088 13904 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39144:
name: "default.Table_2bed983ea7fa4fd39e17cffd5a24225f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:36.036386 13906 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39144:
name: "default.Table_9dd8cc0b642941cfadbb98ab83b7f175"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251212 02:55:36.041428 13965 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1765508123, eventType=CREATE_TABLE, dbName=default, tableName=table_f6ccb5471f934cf0a9ff7e662f65b997, message={"server":"","servicePrincipal":"","db":"default","table":"table_f6ccb5471f934cf0a9ff7e662f65b997","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f6ccb5471f934cf0a9ff7e662f65b997\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765508123},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/warehouse/table_f6ccb5471f934cf0a9ff7e662f65b997\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f6ccb5471f934cf0a9ff7e662f65b997\",\"kudu.table_id\":\"45586ce12b354c4cac23e4a22aafe67c\",\"transient_lastDdlTime\":\"1765508123\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cbec40b29e204890941f676300eb9cd8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765508123,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765508123, eventType=CREATE_TABLE, dbName=default, tableName=table_a63f5d536bfd491fb1d666840afc41a4, message={"server":"","servicePrincipal":"","db":"default","table":"table_a63f5d536bfd491fb1d666840afc41a4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a63f5d536bfd491fb1d666840afc41a4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765508123},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/warehouse/table_a63f5d536bfd491fb1d666840afc41a4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a63f5d536bfd491fb1d666840afc41a4\",\"kudu.table_id\":\"6f9b849174f44f8b94870d02244aeba0\",\"transient_lastDdlTime\":\"1765508123\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cbec40b29e204890941f676300eb9cd8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765508123,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251212 02:55:36.042124 13792 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Refusing update from remote peer 62ed31a99dac451981f4562901f11e87: Log matching property violated. Preceding OpId in replica: term: 5 index: 18. Preceding OpId from leader: term: 7 index: 21. (index mismatch)
W20251212 02:55:36.042981 13898 consensus_peers.cc:597] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 -> Peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Couldn't send request to peer 7823e6e682ac46a09f968e0742506d6e. Status: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251212 02:55:36.043203 13952 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.000s
@ 0x7f46c5535dcd google::LogMessage::Fail() at ??:0
@ 0x7f46c5539b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f46c55357cc google::LogMessage::Flush() at ??:0
@ 0x7f46c5535a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f46dbefcaa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f46dbefb351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251212 02:55:36.062728 13971 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ed31a99dac451981f4562901f11e87. Latest consensus state: current_term: 7 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:36.063009 13971 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:36.066190 13983 mvcc.cc:204] Tried to move back new op lower bound from 7231521325218807808 to 7231521311363690496. Current Snapshot: MvccSnapshot[applied={T|T < 7231521325218807808}]
@ 0x7f46dbf02fcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251212 02:55:36.074666 13912 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Sending DeleteTablet for 3 replicas of tablet ef7a386d14cb4d43b9affd638418d0ca
I20251212 02:55:36.076962 13912 catalog_manager.cc:6724] ReplaceTablet: tablet ef7a386d14cb4d43b9affd638418d0ca deleted and replaced by tablet 2bef99a40c8b4587b3f074b63713e7d0
@ 0x7f46dbf02e49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
I20251212 02:55:36.085325 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:36.085582 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: LEADER
I20251212 02:55:36.086761 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:36.086982 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: LEADER
I20251212 02:55:36.090512 13971 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:36.090716 13971 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:36.091109 13993 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "62ed31a99dac451981f4562901f11e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:36.091315 13993 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:36.097525 12941 tablet_service.cc:1552] Processing DeleteTablet for tablet ef7a386d14cb4d43b9affd638418d0ca with delete_type TABLET_DATA_DELETED (replaced by tablet 2bef99a40c8b4587b3f074b63713e7d0) from {username='slave'} at 127.0.0.1:41488
I20251212 02:55:36.101043 14001 tablet_replica.cc:333] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf: stopping tablet replica
I20251212 02:55:36.101225 14001 raft_consensus.cc:2243] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [term 1 LEADER]: Raft consensus shutting down.
I20251212 02:55:36.101924 14001 raft_consensus.cc:2272] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20251212 02:55:36.103899 14001 ts_tablet_manager.cc:1916] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251212 02:55:36.104542 13232 tablet_service.cc:1552] Processing DeleteTablet for tablet ef7a386d14cb4d43b9affd638418d0ca with delete_type TABLET_DATA_DELETED (replaced by tablet 2bef99a40c8b4587b3f074b63713e7d0) from {username='slave'} at 127.0.0.1:51370
I20251212 02:55:36.107401 13087 tablet_service.cc:1552] Processing DeleteTablet for tablet ef7a386d14cb4d43b9affd638418d0ca with delete_type TABLET_DATA_DELETED (replaced by tablet 2bef99a40c8b4587b3f074b63713e7d0) from {username='slave'} at 127.0.0.1:43672
I20251212 02:55:36.108121 14004 tablet_replica.cc:333] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75: stopping tablet replica
I20251212 02:55:36.108315 14004 raft_consensus.cc:2243] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251212 02:55:36.108557 14004 raft_consensus.cc:2272] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251212 02:55:36.109149 14004 ts_tablet_manager.cc:1916] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251212 02:55:36.111886 14001 ts_tablet_manager.cc:1929] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251212 02:55:36.112000 14001 log.cc:1199] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf: Deleting WAL directory at /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-0/wal/wals/ef7a386d14cb4d43b9affd638418d0ca
I20251212 02:55:36.112452 14001 ts_tablet_manager.cc:1950] T ef7a386d14cb4d43b9affd638418d0ca P 480bcf5c1f394464b302900a89ca5bdf: Deleting consensus metadata
I20251212 02:55:36.114066 14003 tablet_replica.cc:333] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca: stopping tablet replica
I20251212 02:55:36.114264 14003 raft_consensus.cc:2243] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20251212 02:55:36.114480 14003 raft_consensus.cc:2272] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20251212 02:55:36.115082 14003 ts_tablet_manager.cc:1916] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251212 02:55:36.115372 14004 ts_tablet_manager.cc:1929] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251212 02:55:36.115491 14004 log.cc:1199] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75: Deleting WAL directory at /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-1/wal/wals/ef7a386d14cb4d43b9affd638418d0ca
I20251212 02:55:36.115942 14004 ts_tablet_manager.cc:1950] T ef7a386d14cb4d43b9affd638418d0ca P 8240d2efd35f443a9515f0db38f09f75: Deleting consensus metadata
I20251212 02:55:36.117449 13898 catalog_manager.cc:4985] TS 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037): tablet ef7a386d14cb4d43b9affd638418d0ca (table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]) successfully deleted
I20251212 02:55:36.119553 13912 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39144:
name: "default.Table_aeb57448bc884b4ca11ecdd045932bc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:36.120916 14003 ts_tablet_manager.cc:1929] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251212 02:55:36.121073 14003 log.cc:1199] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca: Deleting WAL directory at /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/ts-2/wal/wals/ef7a386d14cb4d43b9affd638418d0ca
I20251212 02:55:36.121104 13899 catalog_manager.cc:4985] TS 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431): tablet ef7a386d14cb4d43b9affd638418d0ca (table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]) successfully deleted
I20251212 02:55:36.121351 13088 tablet_service.cc:1505] Processing CreateTablet for tablet 702cfe1a3e75438ba932d34354ffe0b5 (DEFAULT_TABLE table=default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:36.121812 13088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 702cfe1a3e75438ba932d34354ffe0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:36.122253 13087 tablet_service.cc:1505] Processing CreateTablet for tablet 182044220c484c968b45914e5d037e64 (DEFAULT_TABLE table=default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:36.122314 12940 tablet_service.cc:1505] Processing CreateTablet for tablet 702cfe1a3e75438ba932d34354ffe0b5 (DEFAULT_TABLE table=default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:36.122552 13087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 182044220c484c968b45914e5d037e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:36.122706 12940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 702cfe1a3e75438ba932d34354ffe0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:36.125641 12941 tablet_service.cc:1505] Processing CreateTablet for tablet 182044220c484c968b45914e5d037e64 (DEFAULT_TABLE table=default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:36.125859 14003 ts_tablet_manager.cc:1950] T ef7a386d14cb4d43b9affd638418d0ca P 5f54f877d2864fd48526ff4e6b01b9ca: Deleting consensus metadata
I20251212 02:55:36.126063 12941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 182044220c484c968b45914e5d037e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:36.126968 13086 tablet_service.cc:1505] Processing CreateTablet for tablet 2bef99a40c8b4587b3f074b63713e7d0 (DEFAULT_TABLE table=default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:36.126964 13232 tablet_service.cc:1505] Processing CreateTablet for tablet 182044220c484c968b45914e5d037e64 (DEFAULT_TABLE table=default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:36.127302 13086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bef99a40c8b4587b3f074b63713e7d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:36.127329 13232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 182044220c484c968b45914e5d037e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:36.125636 12942 tablet_service.cc:1505] Processing CreateTablet for tablet 2bef99a40c8b4587b3f074b63713e7d0 (DEFAULT_TABLE table=default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:36.129949 12942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bef99a40c8b4587b3f074b63713e7d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:36.130759 13231 tablet_service.cc:1505] Processing CreateTablet for tablet 702cfe1a3e75438ba932d34354ffe0b5 (DEFAULT_TABLE table=default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:36.131132 13231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 702cfe1a3e75438ba932d34354ffe0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 02:55:36.134960 13899 catalog_manager.cc:4985] TS 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927): tablet ef7a386d14cb4d43b9affd638418d0ca (table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]) successfully deleted
I20251212 02:55:36.136025 13230 tablet_service.cc:1505] Processing CreateTablet for tablet 2bef99a40c8b4587b3f074b63713e7d0 (DEFAULT_TABLE table=default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 02:55:36.136515 13230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bef99a40c8b4587b3f074b63713e7d0. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f46c82aa3f9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f46cb8b56db start_thread at ??:0
I20251212 02:55:36.157892 14008 tablet_bootstrap.cc:492] T 182044220c484c968b45914e5d037e64 P 5f54f877d2864fd48526ff4e6b01b9ca: Bootstrap starting.
I20251212 02:55:36.160632 14008 tablet_bootstrap.cc:654] T 182044220c484c968b45914e5d037e64 P 5f54f877d2864fd48526ff4e6b01b9ca: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:36.162508 14007 tablet_bootstrap.cc:492] T 702cfe1a3e75438ba932d34354ffe0b5 P 480bcf5c1f394464b302900a89ca5bdf: Bootstrap starting.
@ 0x7f46c43e671f clone at ??:0
I20251212 02:55:36.164189 14008 tablet_bootstrap.cc:492] T 182044220c484c968b45914e5d037e64 P 5f54f877d2864fd48526ff4e6b01b9ca: No bootstrap required, opened a new log
I20251212 02:55:36.164335 14008 ts_tablet_manager.cc:1403] T 182044220c484c968b45914e5d037e64 P 5f54f877d2864fd48526ff4e6b01b9ca: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20251212 02:55:36.164660 14007 tablet_bootstrap.cc:654] T 702cfe1a3e75438ba932d34354ffe0b5 P 480bcf5c1f394464b302900a89ca5bdf: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:36.164932 14009 tablet_bootstrap.cc:492] T 182044220c484c968b45914e5d037e64 P 8240d2efd35f443a9515f0db38f09f75: Bootstrap starting.
I20251212 02:55:36.165246 14008 raft_consensus.cc:359] T 182044220c484c968b45914e5d037e64 P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:36.165445 14008 raft_consensus.cc:385] T 182044220c484c968b45914e5d037e64 P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:36.165529 14008 raft_consensus.cc:740] T 182044220c484c968b45914e5d037e64 P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f54f877d2864fd48526ff4e6b01b9ca, State: Initialized, Role: FOLLOWER
I20251212 02:55:36.165798 14008 consensus_queue.cc:260] T 182044220c484c968b45914e5d037e64 P 5f54f877d2864fd48526ff4e6b01b9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:36.166399 13298 heartbeater.cc:499] Master 127.12.17.254:40649 was elected leader, sending a full tablet report...
I20251212 02:55:36.166805 14009 tablet_bootstrap.cc:654] T 182044220c484c968b45914e5d037e64 P 8240d2efd35f443a9515f0db38f09f75: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:36.170652 14008 ts_tablet_manager.cc:1434] T 182044220c484c968b45914e5d037e64 P 5f54f877d2864fd48526ff4e6b01b9ca: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20251212 02:55:36.174633 14008 tablet_bootstrap.cc:492] T 702cfe1a3e75438ba932d34354ffe0b5 P 5f54f877d2864fd48526ff4e6b01b9ca: Bootstrap starting.
I20251212 02:55:36.176707 14008 tablet_bootstrap.cc:654] T 702cfe1a3e75438ba932d34354ffe0b5 P 5f54f877d2864fd48526ff4e6b01b9ca: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:36.177685 14007 tablet_bootstrap.cc:492] T 702cfe1a3e75438ba932d34354ffe0b5 P 480bcf5c1f394464b302900a89ca5bdf: No bootstrap required, opened a new log
I20251212 02:55:36.177889 14007 ts_tablet_manager.cc:1403] T 702cfe1a3e75438ba932d34354ffe0b5 P 480bcf5c1f394464b302900a89ca5bdf: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.008s
I20251212 02:55:36.178382 14009 tablet_bootstrap.cc:492] T 182044220c484c968b45914e5d037e64 P 8240d2efd35f443a9515f0db38f09f75: No bootstrap required, opened a new log
I20251212 02:55:36.178521 14009 ts_tablet_manager.cc:1403] T 182044220c484c968b45914e5d037e64 P 8240d2efd35f443a9515f0db38f09f75: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20251212 02:55:36.178959 14007 raft_consensus.cc:359] T 702cfe1a3e75438ba932d34354ffe0b5 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.179206 14007 raft_consensus.cc:385] T 702cfe1a3e75438ba932d34354ffe0b5 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:36.179309 14007 raft_consensus.cc:740] T 702cfe1a3e75438ba932d34354ffe0b5 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 480bcf5c1f394464b302900a89ca5bdf, State: Initialized, Role: FOLLOWER
I20251212 02:55:36.179252 14009 raft_consensus.cc:359] T 182044220c484c968b45914e5d037e64 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:36.179441 14009 raft_consensus.cc:385] T 182044220c484c968b45914e5d037e64 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:36.179531 14009 raft_consensus.cc:740] T 182044220c484c968b45914e5d037e64 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8240d2efd35f443a9515f0db38f09f75, State: Initialized, Role: FOLLOWER
I20251212 02:55:36.179649 14007 consensus_queue.cc:260] T 702cfe1a3e75438ba932d34354ffe0b5 P 480bcf5c1f394464b302900a89ca5bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.179781 14009 consensus_queue.cc:260] T 182044220c484c968b45914e5d037e64 P 8240d2efd35f443a9515f0db38f09f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:36.180492 13008 heartbeater.cc:499] Master 127.12.17.254:40649 was elected leader, sending a full tablet report...
I20251212 02:55:36.183566 14007 ts_tablet_manager.cc:1434] T 702cfe1a3e75438ba932d34354ffe0b5 P 480bcf5c1f394464b302900a89ca5bdf: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251212 02:55:36.185139 14007 tablet_bootstrap.cc:492] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf: Bootstrap starting.
I20251212 02:55:36.189653 14007 tablet_bootstrap.cc:654] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:36.193964 14007 tablet_bootstrap.cc:492] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf: No bootstrap required, opened a new log
I20251212 02:55:36.194152 14007 ts_tablet_manager.cc:1403] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20251212 02:55:36.195140 14007 raft_consensus.cc:359] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:36.195385 14007 raft_consensus.cc:385] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:36.195499 14007 raft_consensus.cc:740] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 480bcf5c1f394464b302900a89ca5bdf, State: Initialized, Role: FOLLOWER
I20251212 02:55:36.195860 14007 consensus_queue.cc:260] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:36.200289 14007 ts_tablet_manager.cc:1434] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251212 02:55:36.204087 14007 tablet_bootstrap.cc:492] T 2bef99a40c8b4587b3f074b63713e7d0 P 480bcf5c1f394464b302900a89ca5bdf: Bootstrap starting.
I20251212 02:55:36.206089 13154 heartbeater.cc:499] Master 127.12.17.254:40649 was elected leader, sending a full tablet report...
I20251212 02:55:36.206687 14008 tablet_bootstrap.cc:492] T 702cfe1a3e75438ba932d34354ffe0b5 P 5f54f877d2864fd48526ff4e6b01b9ca: No bootstrap required, opened a new log
I20251212 02:55:36.206881 14007 tablet_bootstrap.cc:654] T 2bef99a40c8b4587b3f074b63713e7d0 P 480bcf5c1f394464b302900a89ca5bdf: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:36.206866 14008 ts_tablet_manager.cc:1403] T 702cfe1a3e75438ba932d34354ffe0b5 P 5f54f877d2864fd48526ff4e6b01b9ca: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.023s
I20251212 02:55:36.207319 14009 ts_tablet_manager.cc:1434] T 182044220c484c968b45914e5d037e64 P 8240d2efd35f443a9515f0db38f09f75: Time spent starting tablet: real 0.029s user 0.018s sys 0.009s
I20251212 02:55:36.207710 14008 raft_consensus.cc:359] T 702cfe1a3e75438ba932d34354ffe0b5 P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.207954 14008 raft_consensus.cc:385] T 702cfe1a3e75438ba932d34354ffe0b5 P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:36.208055 14008 raft_consensus.cc:740] T 702cfe1a3e75438ba932d34354ffe0b5 P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f54f877d2864fd48526ff4e6b01b9ca, State: Initialized, Role: FOLLOWER
I20251212 02:55:36.208514 14009 tablet_bootstrap.cc:492] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75: Bootstrap starting.
I20251212 02:55:36.209728 14008 consensus_queue.cc:260] T 702cfe1a3e75438ba932d34354ffe0b5 P 5f54f877d2864fd48526ff4e6b01b9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.210537 14009 tablet_bootstrap.cc:654] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:36.212893 14008 ts_tablet_manager.cc:1434] T 702cfe1a3e75438ba932d34354ffe0b5 P 5f54f877d2864fd48526ff4e6b01b9ca: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20251212 02:55:36.216393 14008 tablet_bootstrap.cc:492] T 2bef99a40c8b4587b3f074b63713e7d0 P 5f54f877d2864fd48526ff4e6b01b9ca: Bootstrap starting.
I20251212 02:55:36.218452 14008 tablet_bootstrap.cc:654] T 2bef99a40c8b4587b3f074b63713e7d0 P 5f54f877d2864fd48526ff4e6b01b9ca: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:36.218598 14007 tablet_bootstrap.cc:492] T 2bef99a40c8b4587b3f074b63713e7d0 P 480bcf5c1f394464b302900a89ca5bdf: No bootstrap required, opened a new log
I20251212 02:55:36.218829 14007 ts_tablet_manager.cc:1403] T 2bef99a40c8b4587b3f074b63713e7d0 P 480bcf5c1f394464b302900a89ca5bdf: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20251212 02:55:36.220009 14007 raft_consensus.cc:359] T 2bef99a40c8b4587b3f074b63713e7d0 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.220435 14007 raft_consensus.cc:385] T 2bef99a40c8b4587b3f074b63713e7d0 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:36.220625 14007 raft_consensus.cc:740] T 2bef99a40c8b4587b3f074b63713e7d0 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 480bcf5c1f394464b302900a89ca5bdf, State: Initialized, Role: FOLLOWER
I20251212 02:55:36.221035 14007 consensus_queue.cc:260] T 2bef99a40c8b4587b3f074b63713e7d0 P 480bcf5c1f394464b302900a89ca5bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.223399 14009 tablet_bootstrap.cc:492] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75: No bootstrap required, opened a new log
I20251212 02:55:36.223573 14009 ts_tablet_manager.cc:1403] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20251212 02:55:36.224426 14009 raft_consensus.cc:359] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.224567 14008 tablet_bootstrap.cc:492] T 2bef99a40c8b4587b3f074b63713e7d0 P 5f54f877d2864fd48526ff4e6b01b9ca: No bootstrap required, opened a new log
I20251212 02:55:36.224643 14009 raft_consensus.cc:385] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:36.224745 14009 raft_consensus.cc:740] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8240d2efd35f443a9515f0db38f09f75, State: Initialized, Role: FOLLOWER
I20251212 02:55:36.224776 14008 ts_tablet_manager.cc:1403] T 2bef99a40c8b4587b3f074b63713e7d0 P 5f54f877d2864fd48526ff4e6b01b9ca: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20251212 02:55:36.224993 14009 consensus_queue.cc:260] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.225795 14009 ts_tablet_manager.cc:1434] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251212 02:55:36.226167 14009 tablet_bootstrap.cc:492] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75: Bootstrap starting.
I20251212 02:55:36.227510 14008 raft_consensus.cc:359] T 2bef99a40c8b4587b3f074b63713e7d0 P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.227705 14008 raft_consensus.cc:385] T 2bef99a40c8b4587b3f074b63713e7d0 P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:36.227792 14008 raft_consensus.cc:740] T 2bef99a40c8b4587b3f074b63713e7d0 P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f54f877d2864fd48526ff4e6b01b9ca, State: Initialized, Role: FOLLOWER
I20251212 02:55:36.227939 14009 tablet_bootstrap.cc:654] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75: Neither blocks nor log segments found. Creating new log.
I20251212 02:55:36.227999 14008 consensus_queue.cc:260] T 2bef99a40c8b4587b3f074b63713e7d0 P 5f54f877d2864fd48526ff4e6b01b9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.225181 14007 ts_tablet_manager.cc:1434] T 2bef99a40c8b4587b3f074b63713e7d0 P 480bcf5c1f394464b302900a89ca5bdf: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251212 02:55:36.230363 14008 ts_tablet_manager.cc:1434] T 2bef99a40c8b4587b3f074b63713e7d0 P 5f54f877d2864fd48526ff4e6b01b9ca: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251212 02:55:36.233942 14009 tablet_bootstrap.cc:492] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75: No bootstrap required, opened a new log
I20251212 02:55:36.234082 14009 ts_tablet_manager.cc:1403] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20251212 02:55:36.234681 14009 raft_consensus.cc:359] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.234838 14009 raft_consensus.cc:385] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 02:55:36.234933 14009 raft_consensus.cc:740] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8240d2efd35f443a9515f0db38f09f75, State: Initialized, Role: FOLLOWER
I20251212 02:55:36.235157 14009 consensus_queue.cc:260] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.235632 14009 ts_tablet_manager.cc:1434] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251212 02:55:36.266968 13978 raft_consensus.cc:493] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:36.267169 13978 raft_consensus.cc:515] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:36.267892 13978 leader_election.cc:290] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037), 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:36.268541 13252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "182044220c484c968b45914e5d037e64" candidate_uuid: "480bcf5c1f394464b302900a89ca5bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" is_pre_election: true
I20251212 02:55:36.268548 13108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "182044220c484c968b45914e5d037e64" candidate_uuid: "480bcf5c1f394464b302900a89ca5bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8240d2efd35f443a9515f0db38f09f75" is_pre_election: true
I20251212 02:55:36.268757 13252 raft_consensus.cc:2468] T 182044220c484c968b45914e5d037e64 P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 480bcf5c1f394464b302900a89ca5bdf in term 0.
I20251212 02:55:36.268764 13108 raft_consensus.cc:2468] T 182044220c484c968b45914e5d037e64 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 480bcf5c1f394464b302900a89ca5bdf in term 0.
I20251212 02:55:36.269253 12897 leader_election.cc:304] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 480bcf5c1f394464b302900a89ca5bdf, 5f54f877d2864fd48526ff4e6b01b9ca, 8240d2efd35f443a9515f0db38f09f75; no voters:
I20251212 02:55:36.269543 13978 raft_consensus.cc:2804] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251212 02:55:36.269662 13978 raft_consensus.cc:493] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 02:55:36.269781 13978 raft_consensus.cc:3060] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:36.271586 13978 raft_consensus.cc:515] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:36.272172 13978 leader_election.cc:290] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [CANDIDATE]: Term 1 election: Requested vote from peers 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037), 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:36.272573 13108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "182044220c484c968b45914e5d037e64" candidate_uuid: "480bcf5c1f394464b302900a89ca5bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8240d2efd35f443a9515f0db38f09f75"
I20251212 02:55:36.272675 13252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "182044220c484c968b45914e5d037e64" candidate_uuid: "480bcf5c1f394464b302900a89ca5bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f54f877d2864fd48526ff4e6b01b9ca"
I20251212 02:55:36.272796 13108 raft_consensus.cc:3060] T 182044220c484c968b45914e5d037e64 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:36.272850 13252 raft_consensus.cc:3060] T 182044220c484c968b45914e5d037e64 P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:36.275607 13108 raft_consensus.cc:2468] T 182044220c484c968b45914e5d037e64 P 8240d2efd35f443a9515f0db38f09f75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 480bcf5c1f394464b302900a89ca5bdf in term 1.
I20251212 02:55:36.275616 13252 raft_consensus.cc:2468] T 182044220c484c968b45914e5d037e64 P 5f54f877d2864fd48526ff4e6b01b9ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 480bcf5c1f394464b302900a89ca5bdf in term 1.
I20251212 02:55:36.276036 12896 leader_election.cc:304] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 480bcf5c1f394464b302900a89ca5bdf, 8240d2efd35f443a9515f0db38f09f75; no voters:
I20251212 02:55:36.276350 13978 raft_consensus.cc:2804] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [term 1 FOLLOWER]: Leader election won for term 1
I20251212 02:55:36.276829 13978 raft_consensus.cc:697] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [term 1 LEADER]: Becoming Leader. State: Replica: 480bcf5c1f394464b302900a89ca5bdf, State: Running, Role: LEADER
I20251212 02:55:36.277218 13978 consensus_queue.cc:237] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } }
I20251212 02:55:36.363411 14016 raft_consensus.cc:493] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:36.363631 14016 raft_consensus.cc:515] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.364400 14016 leader_election.cc:290] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431), 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:36.365106 13252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "702cfe1a3e75438ba932d34354ffe0b5" candidate_uuid: "8240d2efd35f443a9515f0db38f09f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" is_pre_election: true
I20251212 02:55:36.365075 12962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "702cfe1a3e75438ba932d34354ffe0b5" candidate_uuid: "8240d2efd35f443a9515f0db38f09f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "480bcf5c1f394464b302900a89ca5bdf" is_pre_election: true
I20251212 02:55:36.365363 13252 raft_consensus.cc:2468] T 702cfe1a3e75438ba932d34354ffe0b5 P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8240d2efd35f443a9515f0db38f09f75 in term 0.
I20251212 02:55:36.365367 12962 raft_consensus.cc:2468] T 702cfe1a3e75438ba932d34354ffe0b5 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8240d2efd35f443a9515f0db38f09f75 in term 0.
I20251212 02:55:36.365783 13043 leader_election.cc:304] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f54f877d2864fd48526ff4e6b01b9ca, 8240d2efd35f443a9515f0db38f09f75; no voters:
I20251212 02:55:36.366117 14016 raft_consensus.cc:2804] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251212 02:55:36.366233 14016 raft_consensus.cc:493] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 02:55:36.366315 14016 raft_consensus.cc:3060] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:36.368165 14016 raft_consensus.cc:515] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.368618 14016 leader_election.cc:290] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [CANDIDATE]: Term 1 election: Requested vote from peers 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431), 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:36.369114 13252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "702cfe1a3e75438ba932d34354ffe0b5" candidate_uuid: "8240d2efd35f443a9515f0db38f09f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f54f877d2864fd48526ff4e6b01b9ca"
I20251212 02:55:36.369107 12962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "702cfe1a3e75438ba932d34354ffe0b5" candidate_uuid: "8240d2efd35f443a9515f0db38f09f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "480bcf5c1f394464b302900a89ca5bdf"
I20251212 02:55:36.369338 12962 raft_consensus.cc:3060] T 702cfe1a3e75438ba932d34354ffe0b5 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:36.369354 13252 raft_consensus.cc:3060] T 702cfe1a3e75438ba932d34354ffe0b5 P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:36.371089 13252 raft_consensus.cc:2468] T 702cfe1a3e75438ba932d34354ffe0b5 P 5f54f877d2864fd48526ff4e6b01b9ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8240d2efd35f443a9515f0db38f09f75 in term 1.
I20251212 02:55:36.371447 13043 leader_election.cc:304] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f54f877d2864fd48526ff4e6b01b9ca, 8240d2efd35f443a9515f0db38f09f75; no voters:
I20251212 02:55:36.371701 14016 raft_consensus.cc:2804] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [term 1 FOLLOWER]: Leader election won for term 1
I20251212 02:55:36.372145 14016 raft_consensus.cc:697] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [term 1 LEADER]: Becoming Leader. State: Replica: 8240d2efd35f443a9515f0db38f09f75, State: Running, Role: LEADER
I20251212 02:55:36.372490 12962 raft_consensus.cc:2468] T 702cfe1a3e75438ba932d34354ffe0b5 P 480bcf5c1f394464b302900a89ca5bdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8240d2efd35f443a9515f0db38f09f75 in term 1.
I20251212 02:55:36.372499 14016 consensus_queue.cc:237] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
W20251212 02:55:36.380465 13043 connection.cc:537] server connection from 127.0.0.1:43672 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:36.380602 12895 connection.cc:537] client connection to 127.12.17.254:40649 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:36.380658 13185 connection.cc:537] client connection to 127.12.17.254:40649 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:36.380748 13041 connection.cc:537] client connection to 127.12.17.254:40649 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:36.381783 13304 connection.cc:537] client connection to 127.12.17.254:40649 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:36.382521 13304 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.17.254:40649)
W20251212 02:55:36.383086 13304 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.17.254:40649)
W20251212 02:55:36.383409 13304 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.17.254:40649)
W20251212 02:55:36.383489 13865 negotiation.cc:337] Failed RPC negotiation. Trace:
1212 02:55:36.381416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.17.254:40649 (local address 127.12.17.193:47585)
1212 02:55:36.381644 (+ 228us) negotiation.cc:107] Waiting for socket to connect
1212 02:55:36.381726 (+ 82us) client_negotiation.cc:174] Beginning negotiation
1212 02:55:36.381837 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1212 02:55:36.383262 (+ 1425us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":109}
W20251212 02:55:36.383560 13991 negotiation.cc:337] Failed RPC negotiation. Trace:
1212 02:55:36.381403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.17.254:40649 (local address 127.12.17.195:42269)
1212 02:55:36.381644 (+ 241us) negotiation.cc:107] Waiting for socket to connect
1212 02:55:36.381667 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1212 02:55:36.381790 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1212 02:55:36.382483 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126}
W20251212 02:55:36.384238 13866 negotiation.cc:337] Failed RPC negotiation. Trace:
1212 02:55:36.381411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.17.254:40649 (local address 127.12.17.194:57125)
1212 02:55:36.382003 (+ 592us) negotiation.cc:107] Waiting for socket to connect
1212 02:55:36.382022 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1212 02:55:36.382116 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1212 02:55:36.384034 (+ 1918us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":476}
I20251212 02:55:36.531992 14016 raft_consensus.cc:493] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:36.532276 14016 raft_consensus.cc:515] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.532994 14016 leader_election.cc:290] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431), 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:36.533550 12962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2bef99a40c8b4587b3f074b63713e7d0" candidate_uuid: "8240d2efd35f443a9515f0db38f09f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "480bcf5c1f394464b302900a89ca5bdf" is_pre_election: true
I20251212 02:55:36.533653 13252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2bef99a40c8b4587b3f074b63713e7d0" candidate_uuid: "8240d2efd35f443a9515f0db38f09f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" is_pre_election: true
I20251212 02:55:36.533792 12962 raft_consensus.cc:2468] T 2bef99a40c8b4587b3f074b63713e7d0 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8240d2efd35f443a9515f0db38f09f75 in term 0.
I20251212 02:55:36.533872 13252 raft_consensus.cc:2468] T 2bef99a40c8b4587b3f074b63713e7d0 P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8240d2efd35f443a9515f0db38f09f75 in term 0.
I20251212 02:55:36.534240 13043 leader_election.cc:304] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 480bcf5c1f394464b302900a89ca5bdf, 8240d2efd35f443a9515f0db38f09f75; no voters:
I20251212 02:55:36.534536 14016 raft_consensus.cc:2804] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251212 02:55:36.534619 14016 raft_consensus.cc:493] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 02:55:36.534742 14016 raft_consensus.cc:3060] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:36.537168 14016 raft_consensus.cc:515] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.537747 14016 leader_election.cc:290] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [CANDIDATE]: Term 1 election: Requested vote from peers 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431), 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:36.538209 12962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2bef99a40c8b4587b3f074b63713e7d0" candidate_uuid: "8240d2efd35f443a9515f0db38f09f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "480bcf5c1f394464b302900a89ca5bdf"
I20251212 02:55:36.538245 13252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2bef99a40c8b4587b3f074b63713e7d0" candidate_uuid: "8240d2efd35f443a9515f0db38f09f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f54f877d2864fd48526ff4e6b01b9ca"
I20251212 02:55:36.538380 12962 raft_consensus.cc:3060] T 2bef99a40c8b4587b3f074b63713e7d0 P 480bcf5c1f394464b302900a89ca5bdf [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:36.538403 13252 raft_consensus.cc:3060] T 2bef99a40c8b4587b3f074b63713e7d0 P 5f54f877d2864fd48526ff4e6b01b9ca [term 0 FOLLOWER]: Advancing to term 1
I20251212 02:55:36.540303 12962 raft_consensus.cc:2468] T 2bef99a40c8b4587b3f074b63713e7d0 P 480bcf5c1f394464b302900a89ca5bdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8240d2efd35f443a9515f0db38f09f75 in term 1.
I20251212 02:55:36.540719 13252 raft_consensus.cc:2468] T 2bef99a40c8b4587b3f074b63713e7d0 P 5f54f877d2864fd48526ff4e6b01b9ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8240d2efd35f443a9515f0db38f09f75 in term 1.
I20251212 02:55:36.540731 13043 leader_election.cc:304] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 480bcf5c1f394464b302900a89ca5bdf, 8240d2efd35f443a9515f0db38f09f75; no voters:
I20251212 02:55:36.541249 14016 raft_consensus.cc:2804] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [term 1 FOLLOWER]: Leader election won for term 1
I20251212 02:55:36.541440 14016 raft_consensus.cc:697] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [term 1 LEADER]: Becoming Leader. State: Replica: 8240d2efd35f443a9515f0db38f09f75, State: Running, Role: LEADER
I20251212 02:55:36.541729 14016 consensus_queue.cc:237] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 } } peers { permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 } } peers { permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 } }
I20251212 02:55:36.597317 14030 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 62ed31a99dac451981f4562901f11e87)
I20251212 02:55:36.597587 14030 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:36.598281 14030 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:36.600330 13767 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:55:36.600510 13766 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
I20251212 02:55:36.600633 13766 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:36.600865 14030 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251212 02:55:37.150461 14035 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 62ed31a99dac451981f4562901f11e87)
I20251212 02:55:37.150671 14035 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:37.151324 14035 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:37.153539 13766 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:37.153738 13767 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:37.153836 13767 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:37.154047 14035 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251212 02:55:37.274046 13747 master_runner.cc:186] Error getting master registration for 127.12.17.254:40649: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111),
I20251212 02:55:37.274251 13747 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:55:37.337046 14039 consensus_queue.cc:1048] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" member_type: VOTER last_known_addr { host: "127.12.17.194" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 02:55:37.343375 14043 consensus_queue.cc:1048] T 182044220c484c968b45914e5d037e64 P 480bcf5c1f394464b302900a89ca5bdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 02:55:37.421139 14040 consensus_queue.cc:1048] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 02:55:37.425302 14053 consensus_queue.cc:1048] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" member_type: VOTER last_known_addr { host: "127.12.17.195" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 02:55:37.430318 14054 consensus_queue.cc:1048] T 702cfe1a3e75438ba932d34354ffe0b5 P 8240d2efd35f443a9515f0db38f09f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 02:55:37.443583 14054 consensus_queue.cc:1048] T 2bef99a40c8b4587b3f074b63713e7d0 P 8240d2efd35f443a9515f0db38f09f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" member_type: VOTER last_known_addr { host: "127.12.17.193" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 02:55:37.846400 14060 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 62ed31a99dac451981f4562901f11e87)
I20251212 02:55:37.846648 14060 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:37.847344 14060 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:37.849421 13766 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:37.849611 13767 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:37.849851 13767 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:37.850108 14060 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251212 02:55:38.762806 12359 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
/tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.252:44417
--webserver_interface=127.12.17.252
--webserver_port=42635
--builtin_ntp_servers=127.12.17.212:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251212 02:55:38.890362 14078 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 62ed31a99dac451981f4562901f11e87)
I20251212 02:55:38.890594 14078 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:38.891288 14078 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:38.892318 13766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20251212 02:55:38.893517 13767 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:55:38.893889 13766 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
I20251212 02:55:38.894019 13766 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e
I20251212 02:55:38.894268 14078 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251212 02:55:39.114542 14077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251212 02:55:39.114933 14077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251212 02:55:39.115067 14077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251212 02:55:39.124449 14077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251212 02:55:39.124574 14077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251212 02:55:39.124660 14077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251212 02:55:39.124717 14077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251212 02:55:39.135172 14077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.17.212:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.252:44417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.17.252
--webserver_port=42635
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 12 Dec 2025 02:43:20 UTC on 24a791456cd2
build id 9472
ASAN enabled
I20251212 02:55:39.136932 14077 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 02:55:39.139504 14077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 02:55:39.148057 14091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:39.148072 14090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:39.148803 14093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:39.149328 14077 server_base.cc:1047] running on GCE node
I20251212 02:55:39.150149 14077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251212 02:55:39.151633 14077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251212 02:55:39.152849 14077 hybrid_clock.cc:648] HybridClock initialized: now 1765508139152796 us; error 50 us; skew 500 ppm
I20251212 02:55:39.153189 14077 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251212 02:55:39.155372 14077 webserver.cc:492] Webserver started at http://127.12.17.252:42635/ using document root <none> and password file <none>
I20251212 02:55:39.156104 14077 fs_manager.cc:362] Metadata directory not provided
I20251212 02:55:39.156205 14077 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 02:55:39.161928 14077 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20251212 02:55:39.165522 14099 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:39.167130 14077 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251212 02:55:39.167300 14077 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data,/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
uuid: "7823e6e682ac46a09f968e0742506d6e"
format_stamp: "Formatted at 2025-12-12 02:55:21 on dist-test-slave-8cz7"
I20251212 02:55:39.168212 14077 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 02:55:39.188695 14077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 02:55:39.189565 14077 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 02:55:39.189811 14077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 02:55:39.209951 14077 rpc_server.cc:307] RPC server started. Bound to: 127.12.17.252:44417
I20251212 02:55:39.209980 14150 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.17.252:44417 every 8 connection(s)
I20251212 02:55:39.211611 14077 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/info.pb
I20251212 02:55:39.215220 12359 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu as pid 14077
I20251212 02:55:39.219590 14151 sys_catalog.cc:263] Verifying existing consensus state
I20251212 02:55:39.223181 14151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Bootstrap starting.
I20251212 02:55:39.232903 14151 log.cc:826] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Log is configured to *not* fsync() on all Append() calls
I20251212 02:55:39.248344 14151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=10 ignored=0} mutations{seen=5 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251212 02:55:39.248852 14151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Bootstrap complete.
I20251212 02:55:39.254029 14151 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:39.254711 14151 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7823e6e682ac46a09f968e0742506d6e, State: Initialized, Role: FOLLOWER
I20251212 02:55:39.255259 14151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 4.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:39.255985 14155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:39.256230 14155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:39.256916 14151 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: configured and running, proceeding with master startup.
I20251212 02:55:39.269834 14169 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:39.269942 14169 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: loading cluster ID for follower catalog manager: success
I20251212 02:55:39.267354 14077 master_runner.cc:186] Error getting master registration for 127.12.17.254:40649: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111),
I20251212 02:55:39.273082 14077 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:55:39.273954 14169 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: acquiring CA information for follower catalog manager: success
I20251212 02:55:39.275990 14169 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251212 02:55:39.446429 13297 heartbeater.cc:344] Connected to a master server at 127.12.17.252:44417
I20251212 02:55:39.450049 14116 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" instance_seqno: 1765508123473667) as {username='slave'} at 127.12.17.195:36531; Asking this server to re-register.
I20251212 02:55:39.450971 13297 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:39.452704 14116 ts_manager.cc:194] Registered new tserver with Master: 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:39.496959 13153 heartbeater.cc:344] Connected to a master server at 127.12.17.252:44417
I20251212 02:55:39.498690 14116 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" instance_seqno: 1765508122990947) as {username='slave'} at 127.12.17.194:60215; Asking this server to re-register.
I20251212 02:55:39.499327 13153 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:39.500710 14116 ts_manager.cc:194] Registered new tserver with Master: 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
I20251212 02:55:39.647861 13007 heartbeater.cc:344] Connected to a master server at 127.12.17.252:44417
I20251212 02:55:39.649448 14116 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" instance_seqno: 1765508122427264) as {username='slave'} at 127.12.17.193:47117; Asking this server to re-register.
I20251212 02:55:39.650059 13007 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:39.651142 14116 ts_manager.cc:194] Registered new tserver with Master: 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431)
I20251212 02:55:39.684931 14155 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:39.685254 14155 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:39.687202 14155 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
W20251212 02:55:39.688136 14100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:39.690111 14100 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
I20251212 02:55:39.694134 13792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7823e6e682ac46a09f968e0742506d6e" candidate_term: 5 candidate_status { last_received { term: 4 index: 15 } } ignore_live_leader: false dest_uuid: "ad856a8b02154355b5da573e8fffcb16" is_pre_election: true
I20251212 02:55:39.694410 13792 raft_consensus.cc:2368] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7823e6e682ac46a09f968e0742506d6e for earlier term 5. Current term is 7.
I20251212 02:55:39.695168 14100 leader_election.cc:400] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 5 pre-election: Vote denied by peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7823e6e682ac46a09f968e0742506d6e for earlier term 5. Current term is 7.
I20251212 02:55:39.695250 14100 leader_election.cc:403] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20251212 02:55:39.695538 14155 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 4 FOLLOWER]: Advancing to term 7
I20251212 02:55:39.699508 14155 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 7 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7823e6e682ac46a09f968e0742506d6e for earlier term 5. Current term is 7.
I20251212 02:55:40.308902 14183 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:40.309105 14183 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:40.309899 14183 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
I20251212 02:55:40.313400 13792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7823e6e682ac46a09f968e0742506d6e" candidate_term: 8 candidate_status { last_received { term: 4 index: 15 } } ignore_live_leader: false dest_uuid: "ad856a8b02154355b5da573e8fffcb16" is_pre_election: true
I20251212 02:55:40.313737 13792 raft_consensus.cc:2410] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7823e6e682ac46a09f968e0742506d6e for term 8 because replica has last-logged OpId of term: 7 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 15.
W20251212 02:55:40.314054 14100 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
I20251212 02:55:40.314465 14100 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e; no voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:40.314891 14183 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251212 02:55:40.938990 14191 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 62ed31a99dac451981f4562901f11e87)
I20251212 02:55:40.939198 14191 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:40.939944 14191 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:40.942584 13766 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
I20251212 02:55:40.947320 14126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad856a8b02154355b5da573e8fffcb16" candidate_term: 8 candidate_status { last_received { term: 7 index: 22 } } ignore_live_leader: false dest_uuid: "7823e6e682ac46a09f968e0742506d6e" is_pre_election: true
I20251212 02:55:40.947794 14126 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad856a8b02154355b5da573e8fffcb16 in term 7.
I20251212 02:55:40.948396 13767 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87
I20251212 02:55:40.948820 14191 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251212 02:55:40.948922 14191 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 62ed31a99dac451981f4562901f11e87)
I20251212 02:55:40.948987 14191 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 7 FOLLOWER]: Advancing to term 8
I20251212 02:55:40.951167 14191 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:40.951884 14191 leader_election.cc:290] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 election: Requested vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
I20251212 02:55:40.952430 14126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad856a8b02154355b5da573e8fffcb16" candidate_term: 8 candidate_status { last_received { term: 7 index: 22 } } ignore_live_leader: false dest_uuid: "7823e6e682ac46a09f968e0742506d6e"
I20251212 02:55:40.952618 14126 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 7 FOLLOWER]: Advancing to term 8
W20251212 02:55:40.953404 13766 leader_election.cc:336] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
I20251212 02:55:40.954876 14126 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad856a8b02154355b5da573e8fffcb16 in term 8.
I20251212 02:55:40.955297 13767 leader_election.cc:304] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16; no voters: 62ed31a99dac451981f4562901f11e87
I20251212 02:55:40.955580 14191 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 8 FOLLOWER]: Leader election won for term 8
I20251212 02:55:40.956004 14191 raft_consensus.cc:697] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [term 8 LEADER]: Becoming Leader. State: Replica: ad856a8b02154355b5da573e8fffcb16, State: Running, Role: LEADER
I20251212 02:55:40.956436 14191 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 7.22, Last appended by leader: 22, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:40.958527 14197 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad856a8b02154355b5da573e8fffcb16. Latest consensus state: current_term: 8 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:40.958809 14197 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: LEADER
I20251212 02:55:40.959522 14199 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251212 02:55:40.962107 14199 catalog_manager.cc:679] Loaded metadata for table default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c]
I20251212 02:55:40.962590 14199 catalog_manager.cc:679] Loaded metadata for table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]
I20251212 02:55:40.962944 14199 catalog_manager.cc:679] Loaded metadata for table default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e]
I20251212 02:55:40.964735 14199 tablet_loader.cc:96] loaded metadata for tablet 182044220c484c968b45914e5d037e64 (table default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c])
I20251212 02:55:40.965131 14199 tablet_loader.cc:96] loaded metadata for tablet 2bef99a40c8b4587b3f074b63713e7d0 (table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803])
I20251212 02:55:40.965813 14199 tablet_loader.cc:96] loaded metadata for tablet 702cfe1a3e75438ba932d34354ffe0b5 (table default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e])
I20251212 02:55:40.966825 14199 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251212 02:55:40.967798 14199 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:40.967870 14199 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251212 02:55:40.968999 14199 catalog_manager.cc:1514] Loading token signing keys...
I20251212 02:55:40.969844 14199 catalog_manager.cc:6038] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Loaded TSK: 0
I20251212 02:55:40.970670 14199 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251212 02:55:40.971544 14199 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251212 02:55:40.972374 14199 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251212 02:55:41.064640 13782 master_service.cc:954] ReplaceTablet: received request to replace tablet f5f45a962b144612bda2cd788c22da2c from {username='slave'} at 127.0.0.1:33992
I20251212 02:55:41.064785 13781 master_service.cc:954] ReplaceTablet: received request to replace tablet 9bf247fbaba4447c9846db800a856dcd from {username='slave'} at 127.0.0.1:33992
I20251212 02:55:41.065188 13779 master_service.cc:954] ReplaceTablet: received request to replace tablet f5f45a962b144612bda2cd788c22da2c from {username='slave'} at 127.0.0.1:33992
I20251212 02:55:41.065304 13778 master_service.cc:954] ReplaceTablet: received request to replace tablet 182044220c484c968b45914e5d037e64 from {username='slave'} at 127.0.0.1:33992
I20251212 02:55:41.066150 13774 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "default.Table_b815b4e2b12a41a9a3e0d97ab076f666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:41.066614 13775 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33992:
table { table_name: "default.Table_b36fce53138b45fc9fb1a2d7ea2afae7" } new_table_name: "default.Table_2f6246f54e0e491287fe6061611a2ca9" modify_external_catalogs: true
I20251212 02:55:41.067274 13776 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33992:
table { table_name: "default.Table_f6ccb5471f934cf0a9ff7e662f65b997" } new_table_name: "default.Table_aad0d00ededb42ff96b1d27e16607697" modify_external_catalogs: true
I20251212 02:55:41.067329 13773 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "default.Table_9dd8cc0b642941cfadbb98ab83b7f175"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:41.067869 13782 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "default.Table_4f27d7a3eec743a587a57dd44c7c649c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:41.068149 13780 master_service.cc:954] ReplaceTablet: received request to replace tablet 2bef99a40c8b4587b3f074b63713e7d0 from {username='slave'} at 127.0.0.1:33992
I20251212 02:55:41.068482 13781 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "default.Table_70d527cb5d6a46c0b3a5a994b5db9375"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:41.066246 13777 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33992:
table { table_name: "default.Table_050b838a0a9049a39bee3964eb709425" } new_table_name: "default.Table_7325f02ad24a472ba00bf02a83e75f1b" modify_external_catalogs: true
I20251212 02:55:41.069149 13779 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "default.Table_a63f5d536bfd491fb1d666840afc41a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:41.072451 14126 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Refusing update from remote peer ad856a8b02154355b5da573e8fffcb16: Log matching property violated. Preceding OpId in replica: term: 4 index: 15. Preceding OpId from leader: term: 8 index: 24. (index mismatch)
I20251212 02:55:41.073329 14191 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 15, Time since last communication: 0.000s
F20251212 02:55:41.074877 13838 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1765508123, eventType=CREATE_TABLE, dbName=default, tableName=table_f6ccb5471f934cf0a9ff7e662f65b997, message={"server":"","servicePrincipal":"","db":"default","table":"table_f6ccb5471f934cf0a9ff7e662f65b997","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f6ccb5471f934cf0a9ff7e662f65b997\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765508123},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/warehouse/table_f6ccb5471f934cf0a9ff7e662f65b997\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f6ccb5471f934cf0a9ff7e662f65b997\",\"kudu.table_id\":\"45586ce12b354c4cac23e4a22aafe67c\",\"transient_lastDdlTime\":\"1765508123\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cbec40b29e204890941f676300eb9cd8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765508123,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765508123, eventType=CREATE_TABLE, dbName=default, tableName=table_a63f5d536bfd491fb1d666840afc41a4, message={"server":"","servicePrincipal":"","db":"default","table":"table_a63f5d536bfd491fb1d666840afc41a4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a63f5d536bfd491fb1d666840afc41a4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765508123},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/warehouse/table_a63f5d536bfd491fb1d666840afc41a4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a63f5d536bfd491fb1d666840afc41a4\",\"kudu.table_id\":\"6f9b849174f44f8b94870d02244aeba0\",\"transient_lastDdlTime\":\"1765508123\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cbec40b29e204890941f676300eb9cd8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765508123,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f4effda9dcd google::LogMessage::Fail() at ??:0
@ 0x7f4effdadb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f4effda97cc google::LogMessage::Flush() at ??:0
W20251212 02:55:41.081578 13766 consensus_peers.cc:597] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 -> Peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Couldn't send request to peer 62ed31a99dac451981f4562901f11e87. Status: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f4effda9a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f4f16770aa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f4f1676f351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f4f16776fcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f4f16776e49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
I20251212 02:55:41.104929 14213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:41.105322 14213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:41.105870 14202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad856a8b02154355b5da573e8fffcb16. Latest consensus state: current_term: 8 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:41.106241 14202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:41.106729 14212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:41.107106 14212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:41.107896 14208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:41.108157 14204 mvcc.cc:204] Tried to move back new op lower bound from 7231521325412610048 to 7231521295790518272. Current Snapshot: MvccSnapshot[applied={T|T < 7231521325412610048 or (T in {7231521325412610048})}]
I20251212 02:55:41.108239 14208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:41.108422 14201 mvcc.cc:204] Tried to move back new op lower bound from 7231521345824526336 to 7231521345361326080. Current Snapshot: MvccSnapshot[applied={T|T < 7231521345824526336}]
@ 0x7f4f02b1e3f9 kudu::Thread::SuperviseThread() at ??:0
I20251212 02:55:41.111055 14208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:41.111276 14208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f4f061296db start_thread at ??:0
I20251212 02:55:41.113365 13778 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Sending DeleteTablet for 3 replicas of tablet 182044220c484c968b45914e5d037e64
I20251212 02:55:41.113713 14197 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "ad856a8b02154355b5da573e8fffcb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:41.113884 13780 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16: Sending DeleteTablet for 3 replicas of tablet 2bef99a40c8b4587b3f074b63713e7d0
I20251212 02:55:41.113935 14197 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad856a8b02154355b5da573e8fffcb16 [sys.catalog]: This master's current role is: LEADER
@ 0x7f4efec5a71f clone at ??:0
W20251212 02:55:41.282620 13304 connection.cc:537] client connection to 127.12.17.253:46623 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:41.283170 12895 connection.cc:537] client connection to 127.12.17.253:46623 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:41.283752 13304 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.17.253:46623)
W20251212 02:55:41.284178 13304 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.17.253:46623)
W20251212 02:55:41.284514 13304 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.17.253:46623)
I20251212 02:55:41.496081 13155 heartbeater.cc:499] Master 127.12.17.253:46623 was elected leader, sending a full tablet report...
I20251212 02:55:41.547971 12359 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu pid 14077
I20251212 02:55:41.559329 14146 generic_service.cc:196] Checking for leaks (request via RPC)
I20251212 02:55:41.672927 12359 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu with pid 14077
I20251212 02:55:42.454733 12359 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
/tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.252:44417
--webserver_interface=127.12.17.252
--webserver_port=42635
--builtin_ntp_servers=127.12.17.212:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251212 02:55:42.817373 14238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251212 02:55:42.817730 14238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251212 02:55:42.817847 14238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251212 02:55:42.826462 14238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251212 02:55:42.826572 14238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251212 02:55:42.826673 14238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251212 02:55:42.826743 14238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251212 02:55:42.836755 14238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.17.212:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.252:44417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.17.252
--webserver_port=42635
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 12 Dec 2025 02:43:20 UTC on 24a791456cd2
build id 9472
ASAN enabled
I20251212 02:55:42.838519 14238 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 02:55:42.840652 14238 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 02:55:42.850847 14243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:42.850905 14244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:42.851850 14246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:42.852283 14238 server_base.cc:1047] running on GCE node
I20251212 02:55:42.853029 14238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251212 02:55:42.854528 14238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251212 02:55:42.855783 14238 hybrid_clock.cc:648] HybridClock initialized: now 1765508142855735 us; error 38 us; skew 500 ppm
I20251212 02:55:42.856112 14238 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251212 02:55:42.858381 14238 webserver.cc:492] Webserver started at http://127.12.17.252:42635/ using document root <none> and password file <none>
I20251212 02:55:42.859097 14238 fs_manager.cc:362] Metadata directory not provided
I20251212 02:55:42.859186 14238 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 02:55:42.864777 14238 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20251212 02:55:42.867839 14252 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:42.869421 14238 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251212 02:55:42.869599 14238 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data,/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
uuid: "7823e6e682ac46a09f968e0742506d6e"
format_stamp: "Formatted at 2025-12-12 02:55:21 on dist-test-slave-8cz7"
I20251212 02:55:42.870316 14238 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 02:55:42.884754 14238 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 02:55:42.885720 14238 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 02:55:42.885969 14238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 02:55:42.905972 14238 rpc_server.cc:307] RPC server started. Bound to: 127.12.17.252:44417
I20251212 02:55:42.906006 14303 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.17.252:44417 every 8 connection(s)
I20251212 02:55:42.907600 14238 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/info.pb
I20251212 02:55:42.914953 14304 sys_catalog.cc:263] Verifying existing consensus state
I20251212 02:55:42.916024 12359 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu as pid 14238
I20251212 02:55:42.918728 14304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Bootstrap starting.
I20251212 02:55:42.929618 14304 log.cc:826] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Log is configured to *not* fsync() on all Append() calls
I20251212 02:55:42.949321 14304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=15 ignored=0} mutations{seen=13 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251212 02:55:42.949887 14304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Bootstrap complete.
I20251212 02:55:42.956737 14304 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:42.957700 14304 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7823e6e682ac46a09f968e0742506d6e, State: Initialized, Role: FOLLOWER
I20251212 02:55:42.958443 14304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 8.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:42.959388 14308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:42.959774 14308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:42.960436 14304 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: configured and running, proceeding with master startup.
I20251212 02:55:42.973238 13297 heartbeater.cc:344] Connected to a master server at 127.12.17.252:44417
I20251212 02:55:42.975978 14322 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:42.976094 14322 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: loading cluster ID for follower catalog manager: success
I20251212 02:55:42.980602 14322 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: acquiring CA information for follower catalog manager: success
I20251212 02:55:42.975121 14238 master_runner.cc:186] Error getting master registration for 127.12.17.254:40649: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111),
I20251212 02:55:42.982400 14238 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:55:42.982427 14269 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" instance_seqno: 1765508123473667) as {username='slave'} at 127.12.17.195:52549; Asking this server to re-register.
I20251212 02:55:42.983440 13297 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:42.984054 14322 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251212 02:55:42.985307 14269 ts_manager.cc:194] Registered new tserver with Master: 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:43.018834 13153 heartbeater.cc:344] Connected to a master server at 127.12.17.252:44417
I20251212 02:55:43.020289 14269 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" instance_seqno: 1765508122990947) as {username='slave'} at 127.12.17.194:41019; Asking this server to re-register.
I20251212 02:55:43.020987 13153 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:43.022559 14269 ts_manager.cc:194] Registered new tserver with Master: 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
I20251212 02:55:43.183490 13007 heartbeater.cc:344] Connected to a master server at 127.12.17.252:44417
I20251212 02:55:43.184978 14268 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" instance_seqno: 1765508122427264) as {username='slave'} at 127.12.17.193:43223; Asking this server to re-register.
I20251212 02:55:43.185621 13007 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:43.187031 14268 ts_manager.cc:194] Registered new tserver with Master: 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431)
I20251212 02:55:43.458281 14308 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:43.458537 14308 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:43.460366 14308 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
W20251212 02:55:43.461059 14253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:43.462546 14253 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:43.462795 14253 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:43.462917 14253 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e; no voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:43.463305 14308 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251212 02:55:44.212069 14329 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:44.212307 14329 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:44.213025 14329 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
W20251212 02:55:44.217424 14253 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:44.217645 14253 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:44.217754 14253 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e; no voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:44.218025 14329 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251212 02:55:45.184127 14335 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:45.184397 14335 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:45.185264 14335 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
W20251212 02:55:45.190341 14253 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:45.190568 14253 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:45.190658 14253 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e; no voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:45.190954 14335 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251212 02:55:45.292744 12359 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu pid 14238
I20251212 02:55:45.299790 14299 generic_service.cc:196] Checking for leaks (request via RPC)
I20251212 02:55:45.408258 12359 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu with pid 14238
I20251212 02:55:46.333374 12359 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
/tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.252:44417
--webserver_interface=127.12.17.252
--webserver_port=42635
--builtin_ntp_servers=127.12.17.212:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251212 02:55:46.682945 14347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251212 02:55:46.683270 14347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251212 02:55:46.683382 14347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251212 02:55:46.691756 14347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251212 02:55:46.691860 14347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251212 02:55:46.691936 14347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251212 02:55:46.692013 14347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251212 02:55:46.702118 14347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.17.212:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.252:44417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.17.252
--webserver_port=42635
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 12 Dec 2025 02:43:20 UTC on 24a791456cd2
build id 9472
ASAN enabled
I20251212 02:55:46.703819 14347 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 02:55:46.705651 14347 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 02:55:46.714329 14352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:46.714351 14353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:46.715068 14355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:46.716145 14347 server_base.cc:1047] running on GCE node
I20251212 02:55:46.716974 14347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251212 02:55:46.718441 14347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251212 02:55:46.719667 14347 hybrid_clock.cc:648] HybridClock initialized: now 1765508146719618 us; error 48 us; skew 500 ppm
I20251212 02:55:46.720057 14347 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251212 02:55:46.722410 14347 webserver.cc:492] Webserver started at http://127.12.17.252:42635/ using document root <none> and password file <none>
I20251212 02:55:46.723155 14347 fs_manager.cc:362] Metadata directory not provided
I20251212 02:55:46.723258 14347 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 02:55:46.728768 14347 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20251212 02:55:46.731876 14361 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:46.733438 14347 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251212 02:55:46.733592 14347 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data,/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
uuid: "7823e6e682ac46a09f968e0742506d6e"
format_stamp: "Formatted at 2025-12-12 02:55:21 on dist-test-slave-8cz7"
I20251212 02:55:46.734309 14347 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 02:55:46.755672 14347 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 02:55:46.756515 14347 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 02:55:46.756781 14347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 02:55:46.781936 14347 rpc_server.cc:307] RPC server started. Bound to: 127.12.17.252:44417
I20251212 02:55:46.781953 14412 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.17.252:44417 every 8 connection(s)
I20251212 02:55:46.783816 14347 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/info.pb
I20251212 02:55:46.791191 14413 sys_catalog.cc:263] Verifying existing consensus state
I20251212 02:55:46.793918 12359 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu as pid 14347
I20251212 02:55:46.794624 14413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Bootstrap starting.
I20251212 02:55:46.805590 14413 log.cc:826] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Log is configured to *not* fsync() on all Append() calls
I20251212 02:55:46.825340 14413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=15 ignored=0} mutations{seen=13 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251212 02:55:46.825899 14413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Bootstrap complete.
I20251212 02:55:46.830950 14413 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:46.831636 14413 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7823e6e682ac46a09f968e0742506d6e, State: Initialized, Role: FOLLOWER
I20251212 02:55:46.832294 14413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 8.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:46.833235 14417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:46.833606 14417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:46.834447 14413 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: configured and running, proceeding with master startup.
I20251212 02:55:46.847024 14431 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:46.847239 14431 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: loading cluster ID for follower catalog manager: success
I20251212 02:55:46.851562 14431 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: acquiring CA information for follower catalog manager: success
I20251212 02:55:46.858557 14347 master_runner.cc:186] Error getting master registration for 127.12.17.254:40649: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111),
I20251212 02:55:46.864662 14347 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:55:46.864737 14431 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251212 02:55:47.007297 13297 heartbeater.cc:344] Connected to a master server at 127.12.17.252:44417
I20251212 02:55:47.010486 14377 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" instance_seqno: 1765508123473667) as {username='slave'} at 127.12.17.195:58835; Asking this server to re-register.
I20251212 02:55:47.011389 13297 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:47.013675 14377 ts_manager.cc:194] Registered new tserver with Master: 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:47.042466 13153 heartbeater.cc:344] Connected to a master server at 127.12.17.252:44417
I20251212 02:55:47.043947 14377 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" instance_seqno: 1765508122990947) as {username='slave'} at 127.12.17.194:54975; Asking this server to re-register.
I20251212 02:55:47.044569 13153 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:47.045866 14377 ts_manager.cc:194] Registered new tserver with Master: 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
I20251212 02:55:47.205862 13007 heartbeater.cc:344] Connected to a master server at 127.12.17.252:44417
I20251212 02:55:47.207095 14377 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" instance_seqno: 1765508122427264) as {username='slave'} at 127.12.17.193:44681; Asking this server to re-register.
I20251212 02:55:47.207659 13007 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:47.208861 14377 ts_manager.cc:194] Registered new tserver with Master: 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431)
I20251212 02:55:47.242162 14417 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:47.242439 14417 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:47.244427 14417 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
W20251212 02:55:47.245151 14362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:47.246444 14362 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:47.246672 14362 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:47.246791 14362 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e; no voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:47.247150 14417 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251212 02:55:47.963056 14440 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:47.963287 14440 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:47.964253 14440 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
W20251212 02:55:47.967072 14362 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:47.967288 14362 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:47.967413 14362 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e; no voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:47.967648 14440 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251212 02:55:48.615954 14445 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:48.616142 14445 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:48.616902 14445 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
W20251212 02:55:48.620925 14362 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:48.621166 14362 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:48.621304 14362 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e; no voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:48.621572 14445 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251212 02:55:48.977370 12359 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu pid 14347
I20251212 02:55:48.984220 14408 generic_service.cc:196] Checking for leaks (request via RPC)
I20251212 02:55:49.093434 12359 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu with pid 14347
I20251212 02:55:49.874876 12359 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
/tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.252:44417
--webserver_interface=127.12.17.252
--webserver_port=42635
--builtin_ntp_servers=127.12.17.212:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251212 02:55:50.216116 14457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251212 02:55:50.216478 14457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251212 02:55:50.216603 14457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251212 02:55:50.225076 14457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251212 02:55:50.225188 14457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251212 02:55:50.225304 14457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251212 02:55:50.225369 14457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251212 02:55:50.236038 14457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.17.212:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.252:44417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.17.252
--webserver_port=42635
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 12 Dec 2025 02:43:20 UTC on 24a791456cd2
build id 9472
ASAN enabled
I20251212 02:55:50.237782 14457 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 02:55:50.239513 14457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 02:55:50.247666 14466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:50.247877 14465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:50.248893 14468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:50.249718 14457 server_base.cc:1047] running on GCE node
I20251212 02:55:50.250475 14457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251212 02:55:50.251818 14457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251212 02:55:50.253062 14457 hybrid_clock.cc:648] HybridClock initialized: now 1765508150253005 us; error 42 us; skew 500 ppm
I20251212 02:55:50.253489 14457 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251212 02:55:50.255604 14457 webserver.cc:492] Webserver started at http://127.12.17.252:42635/ using document root <none> and password file <none>
I20251212 02:55:50.256373 14457 fs_manager.cc:362] Metadata directory not provided
I20251212 02:55:50.256480 14457 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 02:55:50.262148 14457 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20251212 02:55:50.265120 14474 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:50.266520 14457 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251212 02:55:50.266686 14457 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data,/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
uuid: "7823e6e682ac46a09f968e0742506d6e"
format_stamp: "Formatted at 2025-12-12 02:55:21 on dist-test-slave-8cz7"
I20251212 02:55:50.267388 14457 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 02:55:50.291944 14457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 02:55:50.292867 14457 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 02:55:50.293160 14457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 02:55:50.313382 14457 rpc_server.cc:307] RPC server started. Bound to: 127.12.17.252:44417
I20251212 02:55:50.313405 14525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.17.252:44417 every 8 connection(s)
I20251212 02:55:50.315002 14457 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-2/data/info.pb
I20251212 02:55:50.321802 14526 sys_catalog.cc:263] Verifying existing consensus state
I20251212 02:55:50.325089 14526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Bootstrap starting.
I20251212 02:55:50.325160 12359 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu as pid 14457
I20251212 02:55:50.336125 14526 log.cc:826] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Log is configured to *not* fsync() on all Append() calls
I20251212 02:55:50.356144 14526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=15 ignored=0} mutations{seen=13 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251212 02:55:50.356680 14526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Bootstrap complete.
I20251212 02:55:50.361795 14526 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:50.362468 14526 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7823e6e682ac46a09f968e0742506d6e, State: Initialized, Role: FOLLOWER
I20251212 02:55:50.363104 14526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 8.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:50.363874 14530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:50.364125 14530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:50.364845 14526 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: configured and running, proceeding with master startup.
I20251212 02:55:50.373806 14457 master_runner.cc:186] Error getting master registration for 127.12.17.254:40649: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111),
I20251212 02:55:50.378315 14457 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:55:50.380163 14544 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:50.380316 14544 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: loading cluster ID for follower catalog manager: success
I20251212 02:55:50.384868 14544 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: acquiring CA information for follower catalog manager: success
I20251212 02:55:50.388023 14544 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251212 02:55:50.605367 13297 heartbeater.cc:344] Connected to a master server at 127.12.17.252:44417
I20251212 02:55:50.606477 13153 heartbeater.cc:344] Connected to a master server at 127.12.17.252:44417
I20251212 02:55:50.608306 14491 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" instance_seqno: 1765508122990947) as {username='slave'} at 127.12.17.194:41133; Asking this server to re-register.
I20251212 02:55:50.608346 14490 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" instance_seqno: 1765508123473667) as {username='slave'} at 127.12.17.195:44023; Asking this server to re-register.
I20251212 02:55:50.609016 13153 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:50.609149 13297 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:50.610774 14491 ts_manager.cc:194] Registered new tserver with Master: 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:50.610922 14490 ts_manager.cc:194] Registered new tserver with Master: 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
I20251212 02:55:50.726687 13007 heartbeater.cc:344] Connected to a master server at 127.12.17.252:44417
I20251212 02:55:50.728005 14490 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" instance_seqno: 1765508122427264) as {username='slave'} at 127.12.17.193:49655; Asking this server to re-register.
I20251212 02:55:50.728585 13007 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:50.729807 14490 ts_manager.cc:194] Registered new tserver with Master: 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431)
I20251212 02:55:50.871323 14550 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:50.871613 14550 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:50.873560 14550 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
W20251212 02:55:50.874184 14475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:50.875540 14475 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:50.875762 14475 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:50.875867 14475 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e; no voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:50.876212 14550 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251212 02:55:51.599287 14558 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:51.599499 14558 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:51.600232 14558 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
W20251212 02:55:51.602452 14475 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:51.602646 14475 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:51.602769 14475 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e; no voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:51.603045 14558 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251212 02:55:52.129091 14567 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:52.129288 14567 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:52.129941 14567 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
W20251212 02:55:52.132174 14475 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649): Network error: Client connection negotiation failed: client connection to 127.12.17.254:40649: connect: Connection refused (error 111)
W20251212 02:55:52.132411 14475 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:52.132512 14475 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7823e6e682ac46a09f968e0742506d6e; no voters: 62ed31a99dac451981f4562901f11e87, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:52.132803 14567 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251212 02:55:53.260901 12359 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
/tmp/dist-test-taskaeoKhp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.254:40649
--webserver_interface=127.12.17.254
--webserver_port=35303
--builtin_ntp_servers=127.12.17.212:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251212 02:55:53.597087 14581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251212 02:55:53.597568 14581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251212 02:55:53.597699 14581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251212 02:55:53.605988 14581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251212 02:55:53.606097 14581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251212 02:55:53.606185 14581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251212 02:55:53.606253 14581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251212 02:55:53.616158 14581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.17.212:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:35701
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.17.254:40649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.17.254
--webserver_port=35303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 12 Dec 2025 02:43:20 UTC on 24a791456cd2
build id 9472
ASAN enabled
I20251212 02:55:53.617928 14581 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 02:55:53.619724 14581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 02:55:53.628509 14590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:53.628571 14589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 02:55:53.629236 14592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 02:55:53.629691 14581 server_base.cc:1047] running on GCE node
I20251212 02:55:53.630470 14581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251212 02:55:53.631875 14581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251212 02:55:53.633067 14581 hybrid_clock.cc:648] HybridClock initialized: now 1765508153633021 us; error 34 us; skew 500 ppm
I20251212 02:55:53.633424 14581 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251212 02:55:53.635529 14581 webserver.cc:492] Webserver started at http://127.12.17.254:35303/ using document root <none> and password file <none>
I20251212 02:55:53.636256 14581 fs_manager.cc:362] Metadata directory not provided
I20251212 02:55:53.636345 14581 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 02:55:53.642030 14581 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20251212 02:55:53.644717 14598 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 02:55:53.645978 14581 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251212 02:55:53.646133 14581 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data,/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
uuid: "62ed31a99dac451981f4562901f11e87"
format_stamp: "Formatted at 2025-12-12 02:55:20 on dist-test-slave-8cz7"
I20251212 02:55:53.646797 14581 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 02:55:53.673920 14581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 02:55:53.674791 14581 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 02:55:53.675056 14581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 02:55:53.694670 14581 rpc_server.cc:307] RPC server started. Bound to: 127.12.17.254:40649
I20251212 02:55:53.694692 14649 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.17.254:40649 every 8 connection(s)
I20251212 02:55:53.696261 14581 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/master-0/data/info.pb
I20251212 02:55:53.701045 12359 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaeoKhp/build/asan/bin/kudu as pid 14581
I20251212 02:55:53.703892 14650 sys_catalog.cc:263] Verifying existing consensus state
I20251212 02:55:53.707319 14650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Bootstrap starting.
I20251212 02:55:53.717317 14650 log.cc:826] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Log is configured to *not* fsync() on all Append() calls
I20251212 02:55:53.738472 14650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Bootstrap replayed 1/1 log segments. Stats: ops{read=22 overwritten=0 applied=22 ignored=0} inserts{seen=13 ignored=0} mutations{seen=11 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251212 02:55:53.739224 14650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: Bootstrap complete.
I20251212 02:55:53.745105 14650 raft_consensus.cc:359] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:53.745915 14650 raft_consensus.cc:740] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ed31a99dac451981f4562901f11e87, State: Initialized, Role: FOLLOWER
I20251212 02:55:53.746574 14650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 22, Last appended: 7.22, Last appended by leader: 22, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:53.747478 14654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:53.747787 14654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: This master's current role is: FOLLOWER
I20251212 02:55:53.748576 14650 sys_catalog.cc:565] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [sys.catalog]: configured and running, proceeding with master startup.
I20251212 02:55:53.755090 14669 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:53.755365 14669 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:53.760900 14668 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:53.761087 14668 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: loading cluster ID for follower catalog manager: success
I20251212 02:55:53.762925 14669 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
I20251212 02:55:53.765450 14668 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: acquiring CA information for follower catalog manager: success
I20251212 02:55:53.768708 14668 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251212 02:55:53.780726 14625 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7823e6e682ac46a09f968e0742506d6e" candidate_term: 9 candidate_status { last_received { term: 8 index: 25 } } ignore_live_leader: false dest_uuid: "62ed31a99dac451981f4562901f11e87" is_pre_election: true
I20251212 02:55:53.781581 14625 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7823e6e682ac46a09f968e0742506d6e in term 7.
W20251212 02:55:53.782663 14475 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:53.783936 14475 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e; no voters: ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:53.784328 14669 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251212 02:55:53.784457 14669 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 02:55:53.784543 14669 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 8 FOLLOWER]: Advancing to term 9
I20251212 02:55:53.789706 14669 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:53.790500 14669 leader_election.cc:290] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 election: Requested vote from peers 62ed31a99dac451981f4562901f11e87 (127.12.17.254:40649), ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623)
I20251212 02:55:53.785980 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:55:53.791373 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:55:53.792057 14625 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7823e6e682ac46a09f968e0742506d6e" candidate_term: 9 candidate_status { last_received { term: 8 index: 25 } } ignore_live_leader: false dest_uuid: "62ed31a99dac451981f4562901f11e87"
I20251212 02:55:53.792261 14625 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 7 FOLLOWER]: Advancing to term 9
W20251212 02:55:53.796934 14475 leader_election.cc:336] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:53.797914 14625 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7823e6e682ac46a09f968e0742506d6e in term 9.
I20251212 02:55:53.798399 14475 leader_election.cc:304] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62ed31a99dac451981f4562901f11e87, 7823e6e682ac46a09f968e0742506d6e; no voters: ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:53.798759 14669 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 9 FOLLOWER]: Leader election won for term 9
I20251212 02:55:53.799254 14669 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [term 9 LEADER]: Becoming Leader. State: Replica: 7823e6e682ac46a09f968e0742506d6e, State: Running, Role: LEADER
I20251212 02:55:53.799870 14669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 8.25, Last appended by leader: 25, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:53.803129 14677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7823e6e682ac46a09f968e0742506d6e. Latest consensus state: current_term: 9 leader_uuid: "7823e6e682ac46a09f968e0742506d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } } }
I20251212 02:55:53.803387 14677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e [sys.catalog]: This master's current role is: LEADER
I20251212 02:55:53.804068 14679 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251212 02:55:53.806427 14679 catalog_manager.cc:679] Loaded metadata for table default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c]
I20251212 02:55:53.806876 14679 catalog_manager.cc:679] Loaded metadata for table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803]
I20251212 02:55:53.807212 14679 catalog_manager.cc:679] Loaded metadata for table default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e]
I20251212 02:55:53.809432 14679 tablet_loader.cc:96] loaded metadata for tablet 702cfe1a3e75438ba932d34354ffe0b5 (table default.table_050b838a0a9049a39bee3964eb709425 [id=c45bc9e9db5e40a393a6f4caab827f4e])
I20251212 02:55:53.809705 14679 tablet_loader.cc:96] loaded metadata for tablet 93f4ce2394f94174913669026fabee44 (table default.table_f6ccb5471f934cf0a9ff7e662f65b997 [id=45586ce12b354c4cac23e4a22aafe67c])
I20251212 02:55:53.810132 14679 tablet_loader.cc:96] loaded metadata for tablet e44996fc15da4841a5629209acf27c54 (table default.table_b36fce53138b45fc9fb1a2d7ea2afae7 [id=b85eee00b133494989877915a4814803])
I20251212 02:55:53.810689 14679 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251212 02:55:53.811543 14679 catalog_manager.cc:1269] Loaded cluster ID: cbec40b29e204890941f676300eb9cd8
I20251212 02:55:53.811614 14679 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251212 02:55:53.812742 14679 catalog_manager.cc:1514] Loading token signing keys...
I20251212 02:55:53.813591 14679 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 7823e6e682ac46a09f968e0742506d6e: Loaded TSK: 0
I20251212 02:55:53.814604 14679 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251212 02:55:53.815410 14679 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251212 02:55:53.816205 14679 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251212 02:55:53.903996 14490 master_service.cc:954] ReplaceTablet: received request to replace tablet 9bf247fbaba4447c9846db800a856dcd from {username='slave'} at 127.0.0.1:36370
I20251212 02:55:53.905130 14486 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36370:
table { table_name: "default.Table_050b838a0a9049a39bee3964eb709425" } new_table_name: "default.Table_7325f02ad24a472ba00bf02a83e75f1b" modify_external_catalogs: true
I20251212 02:55:53.905261 14482 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36370:
table { table_name: "default.Table_b36fce53138b45fc9fb1a2d7ea2afae7" } new_table_name: "default.Table_2f6246f54e0e491287fe6061611a2ca9" modify_external_catalogs: true
I20251212 02:55:53.905989 14483 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36370:
table { table_name: "default.Table_f6ccb5471f934cf0a9ff7e662f65b997" } new_table_name: "default.Table_aad0d00ededb42ff96b1d27e16607697" modify_external_catalogs: true
I20251212 02:55:53.906419 14484 master_service.cc:954] ReplaceTablet: received request to replace tablet 2bef99a40c8b4587b3f074b63713e7d0 from {username='slave'} at 127.0.0.1:36370
I20251212 02:55:53.906203 14487 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36370:
name: "default.Table_aeb57448bc884b4ca11ecdd045932bc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:53.906965 14485 master_service.cc:954] ReplaceTablet: received request to replace tablet 182044220c484c968b45914e5d037e64 from {username='slave'} at 127.0.0.1:36370
I20251212 02:55:53.907228 14488 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36370:
name: "default.Table_2a0b5225731647628e4f9ecd5a0ffb2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:53.907658 14485 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36370:
name: "default.Table_a63f5d536bfd491fb1d666840afc41a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:53.907881 14489 master_service.cc:954] ReplaceTablet: received request to replace tablet 9bf247fbaba4447c9846db800a856dcd from {username='slave'} at 127.0.0.1:36370
I20251212 02:55:53.908479 14489 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36370:
name: "default.Table_9dd8cc0b642941cfadbb98ab83b7f175"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:53.908888 14491 master_service.cc:954] ReplaceTablet: received request to replace tablet 9bf247fbaba4447c9846db800a856dcd from {username='slave'} at 127.0.0.1:36370
I20251212 02:55:53.909446 14491 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36370:
name: "default.Table_70d527cb5d6a46c0b3a5a994b5db9375"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:53.907261 14484 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36370:
name: "default.Table_4f27d7a3eec743a587a57dd44c7c649c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 02:55:53.906363 14490 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36370:
name: "default.Table_b815b4e2b12a41a9a3e0d97ab076f666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251212 02:55:53.913141 14543 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1765508123, eventType=CREATE_TABLE, dbName=default, tableName=table_f6ccb5471f934cf0a9ff7e662f65b997, message={"server":"","servicePrincipal":"","db":"default","table":"table_f6ccb5471f934cf0a9ff7e662f65b997","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f6ccb5471f934cf0a9ff7e662f65b997\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765508123},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/warehouse/table_f6ccb5471f934cf0a9ff7e662f65b997\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f6ccb5471f934cf0a9ff7e662f65b997\",\"kudu.table_id\":\"45586ce12b354c4cac23e4a22aafe67c\",\"transient_lastDdlTime\":\"1765508123\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cbec40b29e204890941f676300eb9cd8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765508123,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765508123, eventType=CREATE_TABLE, dbName=default, tableName=table_a63f5d536bfd491fb1d666840afc41a4, message={"server":"","servicePrincipal":"","db":"default","table":"table_a63f5d536bfd491fb1d666840afc41a4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a63f5d536bfd491fb1d666840afc41a4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765508123},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaeoKhp/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765508103625829-12359-0/minicluster-data/warehouse/table_a63f5d536bfd491fb1d666840afc41a4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a63f5d536bfd491fb1d666840afc41a4\",\"kudu.table_id\":\"6f9b849174f44f8b94870d02244aeba0\",\"transient_lastDdlTime\":\"1765508123\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"cbec40b29e204890941f676300eb9cd8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765508123,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fd05af74dcd google::LogMessage::Fail() at ??:0
@ 0x7fd05af78b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd05af747cc google::LogMessage::Flush() at ??:0
@ 0x7fd05af74a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fd07193baa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fd07193a351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fd071941fcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fd071941e49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7fd05dce93f9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd0612f46db start_thread at ??:0
@ 0x7fd059e2571f clone at ??:0
I20251212 02:55:53.997617 13298 heartbeater.cc:344] Connected to a master server at 127.12.17.254:40649
I20251212 02:55:53.997852 13298 heartbeater.cc:499] Master 127.12.17.254:40649 was elected leader, sending a full tablet report...
I20251212 02:55:54.001647 14615 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5f54f877d2864fd48526ff4e6b01b9ca" instance_seqno: 1765508123473667) as {username='slave'} at 127.12.17.195:54439; Asking this server to re-register.
I20251212 02:55:54.002507 13298 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:54.004375 14615 ts_manager.cc:194] Registered new tserver with Master: 5f54f877d2864fd48526ff4e6b01b9ca (127.12.17.195:42927)
I20251212 02:55:54.033882 13154 heartbeater.cc:344] Connected to a master server at 127.12.17.254:40649
I20251212 02:55:54.034113 13154 heartbeater.cc:499] Master 127.12.17.254:40649 was elected leader, sending a full tablet report...
I20251212 02:55:54.036055 14615 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8240d2efd35f443a9515f0db38f09f75" instance_seqno: 1765508122990947) as {username='slave'} at 127.12.17.194:56417; Asking this server to re-register.
I20251212 02:55:54.036736 13154 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:54.038144 14615 ts_manager.cc:194] Registered new tserver with Master: 8240d2efd35f443a9515f0db38f09f75 (127.12.17.194:43037)
W20251212 02:55:54.142038 13305 connection.cc:537] client connection to 127.12.17.252:44417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:54.142076 13042 connection.cc:537] client connection to 127.12.17.252:44417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:54.142418 13186 connection.cc:537] client connection to 127.12.17.252:44417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251212 02:55:54.142756 13305 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.17.252:44417)
W20251212 02:55:54.143369 13305 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.17.252:44417)
W20251212 02:55:54.143620 13305 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.17.252:44417)
I20251212 02:55:54.183807 13008 heartbeater.cc:344] Connected to a master server at 127.12.17.254:40649
I20251212 02:55:54.184046 13008 heartbeater.cc:499] Master 127.12.17.254:40649 was elected leader, sending a full tablet report...
I20251212 02:55:54.185712 14615 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "480bcf5c1f394464b302900a89ca5bdf" instance_seqno: 1765508122427264) as {username='slave'} at 127.12.17.193:47771; Asking this server to re-register.
I20251212 02:55:54.186472 13008 heartbeater.cc:461] Registering TS with master...
I20251212 02:55:54.187850 14615 ts_manager.cc:194] Registered new tserver with Master: 480bcf5c1f394464b302900a89ca5bdf (127.12.17.193:40431)
I20251212 02:55:54.371148 14698 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:54.371357 14698 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:54.373248 14698 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:54.374468 14600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:55:54.375712 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:55:54.375919 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:54.376041 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:54.376408 14698 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:55:54.904656 14704 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:54.904881 14704 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:54.905615 14704 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:54.907755 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:55:54.907924 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:54.908054 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:54.908344 14704 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:55:55.877929 14711 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:55.878149 14711 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:55.878883 14711 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:55.881524 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:55:55.881696 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:55.881904 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:55.882166 14711 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:55:56.042167 12359 master-stress-test.cc:577] Killing test thread 1/25
I20251212 02:55:57.397476 14720 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:57.397692 14720 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:57.398330 14720 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:57.400274 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:55:57.400426 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:55:57.400585 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:57.400877 14720 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:55:58.794250 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:55:58.794422 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:55:59.118088 14729 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:55:59.118431 14729 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:55:59.119259 14729 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:55:59.121150 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:55:59.121438 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:55:59.121573 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:55:59.121857 14729 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:56:01.068807 14736 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:56:01.069015 14736 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:56:01.069657 14736 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:56:01.070627 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20251212 02:56:01.071727 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:56:01.071867 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:56:01.072012 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:56:01.072254 14736 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:56:03.123910 14743 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:56:03.124118 14743 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:56:03.124886 14743 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:56:03.127312 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:56:03.127521 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:56:03.127676 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:56:03.127938 14743 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:56:03.797430 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:56:03.797626 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:56:08.800333 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:56:08.800513 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:56:09.211447 14762 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:56:09.211664 14762 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:56:09.212396 14762 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:56:09.213045 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251212 02:56:09.214365 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:56:09.214524 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:56:09.214640 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:56:09.214910 14762 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:56:11.421805 14768 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:56:11.422024 14768 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:56:11.422680 14768 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:56:11.424751 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:56:11.424908 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:56:11.425050 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:56:11.425343 14768 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:56:13.803100 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:56:13.803304 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:56:18.805766 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:56:18.805976 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:56:19.228148 14791 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:56:19.228351 14791 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:56:19.229074 14791 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:56:19.229770 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251212 02:56:19.231011 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:56:19.231184 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:56:19.231284 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:56:19.231500 14791 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:56:23.809036 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:56:23.809283 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251212 02:56:24.068452 13299 heartbeater.cc:646] Failed to heartbeat to 127.12.17.253:46623 (85 consecutive failures): Network error: Failed to ping master at 127.12.17.253:46623: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:56:24.099233 13155 heartbeater.cc:646] Failed to heartbeat to 127.12.17.253:46623 (85 consecutive failures): Network error: Failed to ping master at 127.12.17.253:46623: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:56:24.314541 13007 heartbeater.cc:646] Failed to heartbeat to 127.12.17.252:44417 (60 consecutive failures): Network error: Failed to ping master at 127.12.17.252:44417: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:56:28.812115 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:56:28.812305 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:56:33.815404 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:56:33.815624 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:56:35.776813 14830 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:56:35.777319 14830 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:56:35.778622 14830 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:56:35.779877 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251212 02:56:35.781603 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:56:35.781771 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:56:35.781881 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:56:35.782192 14830 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:56:38.818650 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:56:38.818882 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:56:43.827898 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:56:43.828083 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:56:48.831053 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:56:48.831261 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:56:53.834350 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:56:53.834542 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:56:55.492326 14876 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:56:55.492564 14876 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:56:55.493480 14876 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:56:55.494576 14600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251212 02:56:55.495891 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:56:55.496140 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:56:55.496299 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:56:55.496572 14876 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:56:58.837495 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:56:58.837671 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:57:03.840505 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:57:03.840688 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:57:08.843293 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:57:08.843492 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:57:09.275707 14918 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:57:09.275905 14918 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:57:09.276539 14918 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:57:09.277226 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251212 02:57:09.278589 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:57:09.278738 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:57:09.278849 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:57:09.279094 14918 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:57:13.845880 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:57:13.846082 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:57:18.848524 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:57:18.848711 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:57:23.851740 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:57:23.851922 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:57:23.895354 14956 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:57:23.895581 14956 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:57:23.896400 14956 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:57:23.897053 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251212 02:57:23.898294 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:57:23.898492 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:57:23.898626 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:57:23.898885 14956 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20251212 02:57:24.222674 13299 heartbeater.cc:646] Failed to heartbeat to 127.12.17.253:46623 (205 consecutive failures): Network error: Failed to ping master at 127.12.17.253:46623: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:57:24.244866 13155 heartbeater.cc:646] Failed to heartbeat to 127.12.17.253:46623 (205 consecutive failures): Network error: Failed to ping master at 127.12.17.253:46623: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:57:24.464567 13007 heartbeater.cc:646] Failed to heartbeat to 127.12.17.252:44417 (180 consecutive failures): Network error: Failed to ping master at 127.12.17.252:44417: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:57:24.508534 14961 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:57:24.508703 14961 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:57:24.509369 14961 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:57:24.511273 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:57:24.511528 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:57:24.511633 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:57:24.511926 14961 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:57:28.854696 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:57:28.854856 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:57:33.857689 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:57:33.857896 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:57:35.336990 14989 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:57:35.337244 14989 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:57:35.338070 14989 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:57:35.339131 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251212 02:57:35.340225 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:57:35.340391 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:57:35.340596 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:57:35.340929 14989 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:57:38.860872 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:57:38.861068 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:57:43.864146 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:57:43.864352 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:57:48.867201 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:57:48.867412 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:57:48.895862 15029 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:57:48.896131 15029 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:57:48.896982 15029 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:57:48.897753 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251212 02:57:48.899255 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:57:48.899422 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:57:48.899528 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:57:48.899786 15029 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:57:51.237599 15038 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:57:51.237813 15038 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:57:51.238515 15038 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:57:51.240511 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:57:51.240645 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:57:51.240785 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:57:51.241035 15038 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:57:53.857872 14604 maintenance_manager.cc:419] P 62ed31a99dac451981f4562901f11e87: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033364
I20251212 02:57:53.890457 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:57:53.890684 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:57:53.907871 14603 maintenance_manager.cc:643] P 62ed31a99dac451981f4562901f11e87: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.049s user 0.019s sys 0.005s Metrics: {"bytes_written":9116,"cfile_init":1,"dirs.queue_time_us":383,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":689,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":589,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":13,"thread_start_us":466,"threads_started":2,"wal-append.queue_time_us":302}
I20251212 02:57:53.909086 14604 maintenance_manager.cc:419] P 62ed31a99dac451981f4562901f11e87: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 776 bytes on disk
I20251212 02:57:53.909828 14603 maintenance_manager.cc:643] P 62ed31a99dac451981f4562901f11e87: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20251212 02:57:57.665570 15057 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:57:57.665800 15057 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:57:57.666591 15057 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:57:57.667613 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251212 02:57:57.668951 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:57:57.669153 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:57:57.669353 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:57:57.669617 15057 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:57:58.893834 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:57:58.894029 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:57:59.464213 15066 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:57:59.464395 15066 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:57:59.465147 15066 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:57:59.467252 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:57:59.467424 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:57:59.467566 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:57:59.467809 15066 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:58:03.897017 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:58:03.897192 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:58:08.900184 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:58:08.900377 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:58:13.903116 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:58:13.903301 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:58:16.828318 15173 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:58:16.828613 15173 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:58:16.829718 15173 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:58:16.830698 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251212 02:58:16.831941 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:58:16.832144 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:58:16.832247 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:58:16.832484 15173 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:58:18.612224 15186 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:58:18.612422 15186 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:58:18.613070 15186 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:58:18.614917 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:58:18.615051 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:58:18.615216 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:58:18.615501 15186 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:58:18.905897 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:58:18.906078 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:58:23.908973 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:58:23.909145 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251212 02:58:24.369123 13299 heartbeater.cc:646] Failed to heartbeat to 127.12.17.253:46623 (325 consecutive failures): Network error: Failed to ping master at 127.12.17.253:46623: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:58:24.389631 13155 heartbeater.cc:646] Failed to heartbeat to 127.12.17.253:46623 (325 consecutive failures): Network error: Failed to ping master at 127.12.17.253:46623: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:58:24.613109 13007 heartbeater.cc:646] Failed to heartbeat to 127.12.17.252:44417 (300 consecutive failures): Network error: Failed to ping master at 127.12.17.252:44417: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:58:28.911832 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:58:28.912016 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:58:30.513247 15258 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:58:30.513463 15258 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:58:30.514183 15258 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:58:30.515188 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251212 02:58:30.516391 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:58:30.516552 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:58:30.516686 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:58:30.516911 15258 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:58:33.914820 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:58:33.915012 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:58:37.055420 15301 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:58:37.055645 15301 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:58:37.056402 15301 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:58:37.057482 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251212 02:58:37.058760 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:58:37.058934 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:58:37.059137 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:58:37.059377 15301 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:58:38.918057 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:58:38.918287 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:58:43.920967 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:58:43.921164 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:58:48.923866 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:58:48.924091 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:58:49.933117 15380 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:58:49.933359 15380 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:58:49.934110 15380 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:58:49.935360 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251212 02:58:49.936820 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:58:49.937057 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:58:49.937182 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:58:49.937467 15380 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:58:52.020448 15393 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:58:52.020783 15393 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:58:52.021485 15393 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:58:52.026551 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:58:52.026782 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:58:52.026886 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:58:52.027154 15393 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:58:53.929593 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:58:53.929790 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:58:58.932786 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:58:58.932953 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:59:03.935961 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:59:03.936133 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:59:08.938944 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:59:08.939106 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:59:10.552330 15500 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:59:10.552582 15500 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:59:10.553485 15500 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:59:10.554562 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251212 02:59:10.555593 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:59:10.555719 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:59:10.555862 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:59:10.556118 15500 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:59:13.942126 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:59:13.942335 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:59:18.945443 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:59:18.945664 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:59:21.445173 15569 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:59:21.445417 15569 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:59:21.446151 15569 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:59:21.447178 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251212 02:59:21.448676 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:59:21.448885 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:59:21.449016 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:59:21.449363 15569 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:59:23.948417 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:59:23.948601 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251212 02:59:24.516530 13299 heartbeater.cc:646] Failed to heartbeat to 127.12.17.253:46623 (445 consecutive failures): Network error: Failed to ping master at 127.12.17.253:46623: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:59:24.535228 13155 heartbeater.cc:646] Failed to heartbeat to 127.12.17.253:46623 (445 consecutive failures): Network error: Failed to ping master at 127.12.17.253:46623: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:59:24.765853 13007 heartbeater.cc:646] Failed to heartbeat to 127.12.17.252:44417 (420 consecutive failures): Network error: Failed to ping master at 127.12.17.252:44417: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:59:28.951567 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:59:28.951784 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:59:33.954941 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:59:33.955139 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:59:38.958423 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:59:38.958612 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:59:39.009534 15670 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:59:39.009756 15670 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:59:39.010700 15670 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:59:39.011217 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251212 02:59:39.012938 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:59:39.013090 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:59:39.013266 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:59:39.013481 15670 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:59:43.961961 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:59:43.962183 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:59:48.964927 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:59:48.965099 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:59:50.837080 15732 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:59:50.837318 15732 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:59:50.838018 15732 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:59:50.839020 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251212 02:59:50.840248 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 02:59:50.840431 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 02:59:50.840524 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:59:50.840744 15732 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:59:51.566908 15739 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:59:51.567106 15739 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:59:51.567724 15739 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:59:51.570875 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:59:51.571040 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:59:51.571208 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:59:51.571496 15739 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:59:53.967830 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:59:53.968005 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 02:59:54.206650 15757 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 02:59:54.206868 15757 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 02:59:54.207629 15757 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 02:59:54.209403 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 02:59:54.209659 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 02:59:54.209802 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 02:59:54.210035 15757 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 02:59:58.970775 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 02:59:58.970938 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 03:00:03.974144 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 03:00:03.974329 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 03:00:08.977337 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 03:00:08.977514 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 03:00:12.509770 15840 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 03:00:12.509958 15840 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 03:00:12.510915 15840 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 03:00:12.511878 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20251212 03:00:12.512979 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 03:00:12.513130 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 03:00:12.513340 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 03:00:12.513581 15840 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 03:00:13.980235 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 03:00:13.980401 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 03:00:18.983608 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 03:00:18.983791 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 03:00:23.986915 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 03:00:23.987095 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251212 03:00:24.664822 13299 heartbeater.cc:646] Failed to heartbeat to 127.12.17.253:46623 (565 consecutive failures): Network error: Failed to ping master at 127.12.17.253:46623: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 03:00:24.680419 13155 heartbeater.cc:646] Failed to heartbeat to 127.12.17.253:46623 (565 consecutive failures): Network error: Failed to ping master at 127.12.17.253:46623: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 03:00:24.916316 13007 heartbeater.cc:646] Failed to heartbeat to 127.12.17.252:44417 (540 consecutive failures): Network error: Failed to ping master at 127.12.17.252:44417: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 03:00:26.444861 15904 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 03:00:26.445112 15904 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 03:00:26.445873 15904 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 03:00:26.446714 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251212 03:00:26.448282 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
W20251212 03:00:26.448446 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
I20251212 03:00:26.448632 14600 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 03:00:26.448921 15904 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 03:00:28.989745 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 03:00:28.989915 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 03:00:29.702229 15925 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 03:00:29.702418 15925 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 03:00:29.703019 15925 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 03:00:29.704953 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 03:00:29.705123 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 03:00:29.705271 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 03:00:29.705482 15925 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 03:00:33.992913 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 03:00:33.993084 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 03:00:36.163995 14604 maintenance_manager.cc:419] P 62ed31a99dac451981f4562901f11e87: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 776 bytes on disk
I20251212 03:00:36.169294 14603 maintenance_manager.cc:643] P 62ed31a99dac451981f4562901f11e87: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.005s sys 0.000s Metrics: {"dirs.queue_time_us":371,"thread_start_us":484,"threads_started":2,"wal-append.queue_time_us":362}
I20251212 03:00:38.996049 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 03:00:38.996249 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 03:00:43.999068 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 03:00:43.999255 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 03:00:47.667021 16014 raft_consensus.cc:493] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 03:00:47.667258 16014 raft_consensus.cc:515] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed31a99dac451981f4562901f11e87" member_type: VOTER last_known_addr { host: "127.12.17.254" port: 40649 } } peers { permanent_uuid: "ad856a8b02154355b5da573e8fffcb16" member_type: VOTER last_known_addr { host: "127.12.17.253" port: 46623 } } peers { permanent_uuid: "7823e6e682ac46a09f968e0742506d6e" member_type: VOTER last_known_addr { host: "127.12.17.252" port: 44417 } }
I20251212 03:00:47.668164 16014 leader_election.cc:290] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623), 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417)
W20251212 03:00:47.669266 14599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251212 03:00:47.670316 14600 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7823e6e682ac46a09f968e0742506d6e (127.12.17.252:44417): Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 03:00:47.670502 14599 leader_election.cc:336] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad856a8b02154355b5da573e8fffcb16 (127.12.17.253:46623): Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111)
I20251212 03:00:47.670622 14599 leader_election.cc:304] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62ed31a99dac451981f4562901f11e87; no voters: 7823e6e682ac46a09f968e0742506d6e, ad856a8b02154355b5da573e8fffcb16
I20251212 03:00:47.670872 16014 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 62ed31a99dac451981f4562901f11e87 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251212 03:00:49.001945 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 03:00:49.002135 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251212 03:00:54.005236 14581 master_runner.cc:186] Error getting master registration for 127.12.17.253:46623: Network error: Client connection negotiation failed: client connection to 127.12.17.253:46623: connect: Connection refused (error 111),
I20251212 03:00:54.005451 14581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251212 03:00:54.646966 13304 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 03:00:54.647308 13304 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
W20251212 03:00:54.647485 13304 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
F20251212 03:00:54.661307 13333 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1765508454 (unix time) try "date -d @1765508454" if you are using GNU date ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663008 13337 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663008 13337 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663182 13317 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2a0b5225731647628e4f9ecd5a0ffb2c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663206 1*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663008 13337 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663182 13317 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2a0b5225731647628e4f9ecd5a0ffb2c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663206 13336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663008 13337 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663182 13317 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2a0b5225731647628e4f9ecd5a0ffb2c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663206 13336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663276 13323 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663008 13337 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663182 13317 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2a0b5225731647628e4f9ecd5a0ffb2c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663206 13336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663276 13323 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664569 13314 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_aeb57448bc884b4ca11ecdd045932bc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663008 13337 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663182 13317 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2a0b5225731647628e4f9ecd5a0ffb2c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663206 13336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663276 13323 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664569 13314 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_aeb57448bc884b4ca11ecdd045932bc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664646 13313 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a63f5d536bfd491fb1d666840afc41a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663008 13337 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663182 13317 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2a0b5225731647628e4f9ecd5a0ffb2c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663206 13336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663276 13323 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664569 13314 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_aeb57448bc884b4ca11ecdd045932bc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664646 13313 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a63f5d536bfd491fb1d666840afc41a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.665846 13322 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663008 13337 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663182 13317 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2a0b5225731647628e4f9ecd5a0ffb2c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663206 13336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663276 13323 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664569 13314 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_aeb57448bc884b4ca11ecdd045932bc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664646 13313 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a63f5d536bfd491fb1d666840afc41a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.665846 13322 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.665915 13332 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663008 13337 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663182 13317 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2a0b5225731647628e4f9ecd5a0ffb2c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663206 13336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663276 13323 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664569 13314 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_aeb57448bc884b4ca11ecdd045932bc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664646 13313 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a63f5d536bfd491fb1d666840afc41a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.665846 13322 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.665915 13332 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.666059 13338 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663008 13337 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663182 13317 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2a0b5225731647628e4f9ecd5a0ffb2c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663206 13336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663276 13323 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664569 13314 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_aeb57448bc884b4ca11ecdd045932bc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664646 13313 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a63f5d536bfd491fb1d666840afc41a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.665846 13322 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.665915 13332 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.666059 13338 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.666235 13312 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_70d527cb5d6a46c0b3a5a994b5db9375 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663008 13337 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663182 13317 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2a0b5225731647628e4f9ecd5a0ffb2c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663206 13336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663276 13323 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664569 13314 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_aeb57448bc884b4ca11ecdd045932bc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664646 13313 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a63f5d536bfd491fb1d666840afc41a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.665846 13322 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.665915 13332 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.666059 13338 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.666235 13312 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_70d527cb5d6a46c0b3a5a994b5db9375 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.666288 13316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fc25d2c6f4ad42958441b5b6c7eab15c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663008 13337 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663182 13317 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2a0b5225731647628e4f9ecd5a0ffb2c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663206 13336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663276 13323 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664569 13314 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_aeb57448bc884b4ca11ecdd045932bc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664646 13313 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a63f5d536bfd491fb1d666840afc41a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.665846 13322 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.665915 13332 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.666059 13338 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.666235 13312 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_70d527cb5d6a46c0b3a5a994b5db9375 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.666288 13316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fc25d2c6f4ad42958441b5b6c7eab15c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.667881 13320 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dd8cc0b642941cfadbb98ab83b7f175 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
PC: @ 0x0 (unknown)
*** Check failure stack trace: ***
F20251212 03:00:54.661604 13315 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2bed983ea7fa4fd39e17cffd5a24225f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661729 13324 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661769 13319 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_397fb5c3eafe4e49b3667efcf7811f7a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.661972 13321 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b815b4e2b12a41a9a3e0d97ab076f666 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663008 13337 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663182 13317 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2a0b5225731647628e4f9ecd5a0ffb2c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663206 13336 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.663276 13323 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664569 13314 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_aeb57448bc884b4ca11ecdd045932bc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.664646 13313 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a63f5d536bfd491fb1d666840afc41a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.665846 13322 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.665915 13332 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.666059 13338 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.666235 13312 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_70d527cb5d6a46c0b3a5a994b5db9375 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.666288 13316 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fc25d2c6f4ad42958441b5b6c7eab15c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.667881 13320 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dd8cc0b642941cfadbb98ab83b7f175 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)F20251212 03:00:54.667964 13318 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4f27d7a3eec743a587a57dd44c7c649c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.17.254:40649,127.12.17.253:46623,127.12.17.252:44417, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.17.252:44417: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** SIGABRT (@0x3e800003047) received by PID 12359 (TID 0x7f17541aa700) from PID 12359; stack trace: ***
@ 0x7f177160f980 (unknown) at ??:0
@ 0x7f176a3e4fb7 gsignal at ??:0
@ 0x7f176a3e6921 abort at ??:0
@ 0x7f176d11cdcd google::LogMessage::Fail() at ??:0
@ 0x7f176d120b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f176d11c7cc google::LogMessage::Flush() at ??:0
@ 0x7f176d11df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40d9f5 kudu::MasterStressTest::ReplaceTabletThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:391
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7f176aa6c6df (unknown) at ??:0
@ 0x7f17716046db start_thread at ??:0
@ 0x7f176a4c771f clone at ??:0